Design And Implementation Of Mass Data Analysis System Based On Hadoop

With the continuous development of the Internet, business of a large domesticcompany is going to be more and more complex. Analysis of core data becomes a keypoint of the development of the company, but the internal core raw data files are huge.They get the conclusion by manual analysis. Therefore, analysis and processingmassive data become problems to be solved.In this paper, we research the home and ab road status for massive dataprocessing. The MapReduce distributed programming idea is elaborated. Weintroduce technologies about Hadoop and the HDFS file system, analysis of the needsof enterprises. Spirng-Mvc and Hibernate web development framework are applied tohierarchical designing in the system. The system is divided into five layers: viewlayer，business logic layer，data object layer，underlying data layer and originalresource layer. Then we expand the design and implementation of the systemstructure. Hadoop calculation module, data storage module and business systemmodule are designed with actual business. Finally, each module is tested, Mass dataanalysis system is finished.The work in this paper makes use of the development of a system of a largeInternet company. Practice shows that the system designed in this paper improves theefficiency of data analysis，it changes the status of artificially calculating the massdata and makes the statistical data analysis efficient and centralized.