Design And Implementation Of Telecom Data Analysis Based On Hadoop

The domestic mobile Internet has entered a stage of rapid development in 2010. But with the Internet company based intervention and terminal manufacturers to quickly join the application store model,telecom operators appear plight data revenue slowdown, the threat is pipelined. At the same time, along with the popularity of mobile Internet applications,telecom operators datasize from GB to TB or PB. In the commercial competition, run by the auxiliary data analysis has become an effective tool, but the traditional data analysis infrastructure can not meet the demand for such massive data processing and rapid, deep mining. This Hadoop data processing framework for solving the above problems and provides a new way of thinking.Design and implementation of this system as a platform for building large data Shaanxi Telecom pre-research project in this context.Construction of Hadoop system based on exploration,feasibility analysis and mining on the dailycleaning,ten billion data through offline Hadoop platform;Build BI system analysis simulation packages by the above process data traffic packages optimized design;Achieve user analysis system constructed from the access,search,call duration,SMS usage and other acts,multi-dimensional positioning user preferences and interests to form a customer portraits;Establish decision-making system for telecommunications services.This paper analyzes the Hadoop framework and the techniques used in HDFS and MapReduce, then on the Hadoop platform for data acquisition, data storage elaborated; Focuses on the use of MapReduce technology for parallel computing method; After the data stored in the data processing among the HDFS file system, and will be finished by Sqoop assembly process dump relational database;BI design of this system using J2 EE development framework, and detailed design, completed on the basis of background data processing on traffic monitoring, operational support functions, customer-portrait function, decision support function in the realization of the use of clustering algorithms.Configure test environment in the laboratory environment,and separately for data transmission of large data sets and offline data processing under Hadoop,and front-end BI show.System operating normally effective,based on the experimental aspects of show Hadoop platform to meet the basic telecommunications data preprocessing and data storage needs.