Platform Development On Massive Data Collection And Processing Based On Hadoop

With the integration of the mobile network and the Internet, different kinds of data service used by users have become the main way of information transfer. Those service data is transferred over the Internet by the way of IP datagram. At present, the network quality indicators based on NMS can not take control of service effectively according to the characteristics of user behavior or reflect the real user experience of various service. In this case, we need collect IP packet continuously, and then study the analysis system of user behavior characteristics, the law of data service, improve the predictive ability of the network about the user characteristics and promote the development of future network.Network packet capture is the core of this demand and is of great significance to follow-up analysis of data and the characteristics of user behavior.With the beginning of the network data collection, massive data rapidly emerges. It is a servere test to the resources of database servers. With the rapid increase of data resources, all data analysis and processing job to be completed by a single database system alone can not meet the actual needs. Therefore, we need to enhance capabilities of data processing to meet the data processing requirements of large data environment.Accuracy of the data analysis can reflect the value of the data and is good for the study of user behavior characteristics. Therefore, the study of the characteristics of the Internet data can help to portray the behavior of the network accurately and give guidance to the practical network deployment and traffic control, promoting the study of service-oriented future Internet architecture and mechanism.In this paper, we do our research on the areas metioned above, which include:(1) technology of high-speed link packet capture,(2) technology of massive data storage,(3) technology of massive data analysis and (4) data analysis and presentation.