Massive Data Processing Of Mobile Operator Based On Hadoop

The data amount is growing quickly with the mobile Internet development. The traditional telecommunication operators face severe pressure by this trend. The operator income is not increased with the data amount increasing. The telecommunication operators face the challenging that the added-value of data flow is taken off by Internet operators. At the same time, telecommunication operators are possible to become communication channel. For traditional telecommunication operators, the problem is how to use the big data in hand to make profit.In order to meet the challenges of the era of big data, and to take advantage of system data resources, the China Mobile plans to launch a mobile precise marketing platform project to improve the Internet operation support capabilities, which is based on big data technology in terms of customer labels, traffic operations, stock ownership, terminal sales and channel synergies etc. Based on previous reasons, this thesis studies the big data processing for mobile precise marketing platform, which mainly focuses on data gathering and pre-processing, and big data security.This thesis investigates the data warehouse technology(ETL, Extract-Transform-Load), the metadata, common warehouse meta-model(CWM, Common Warehouse Meta-model) and related knowledge of Hadoop. Based on MapReduce parallel processing framework and Mpp, and cloud-based ETL and crawler technology, the data mining and data pre-processing functions are designed for mobile precise operation platform. The data security-related functions, and metadata model of the operation system, are designed in this thesis.