Research And Implementation Of Distributed Cube Distributed Storage And Construction Algorithm

With the Big Data Era is coming, the analysis of big data become the development bottleneck, which has exist for several years in the field of Military, Financial and Communication. With the rapid development of Internet and Information industry, this problem becomes more urgent to be resolved. At present, it could be alleviated by the pretreatment, while these only could short the query time for customer and save the disk pace, this problem has not yet been fundamentally resolved. Challenged with the big data application needs, the efficient visit, comprehensive analysis and effective management about big data become an urgent problem to be resolved.Based on the comprehensive analysis of MapReduce parallel computing architecture, Spark data compute platform, BTS algorithm and DFS algorithm, the generation algorithm of closed data cube were study in this paper:Based on the improved BTS algorithm from MapReduce and the improved DFS algorithm from Spark big data computing platform, a closed data cube were constructed; Two BTS algorithm were improved by generating complete closed cube and local closed cube respectively, the generating efficiency of each closed cube were effectively raised by the distributed computing style; The iterative style calculate fit for the Spark platform, so the DFS algorithm was improved, which could effectively raise the generation efficiency of closed cube also. Finally closed cube query algorithm and incremental maintenance method were analyzed, the correctness and stability were verified by the simulation experiment.The efficiency of closed cube generation and data warehouse building with big data platform would be raised by the research results, which were expected to provide new ideas for resolving the big data problem in all fields.