Research And Implementation Of Mass Network Traffic Analysis System Based On Time Series

In recent years,with the rapid development of the Internet,a variety of network applications springs up all over,Chinese and even the world's Internet user are growing.According to the thirty-sixth China Internet development statistics report released by CCNIC,as of June 2015,the scale of China's Internet users reached 668000000,half a year a total of 18.94 million Internet users were created.Internet penetration rate of 48.8%.With the increasing popularity of the Internet,the connection between people in daily life and the Internet is becoming more and more closely,the Internet generated by the network traffic data is also growing.For these vast amounts of network traffic data containing personal information,how it is processed,analyzed and stored in the operator concerned is a major problem.To some extent,the emergence of Hadoop has solved the problem of mass network traffic data processing,but the new problems arise.The batch processing mechanism of Hadoop makes it more suitable for processing massive amounts of data,while the performance of real-time data processing is unsatisfactory.And the appearance of the Storm stream processing platform has just made up for the shortcomings in real-time processing of the Hadoop processing platform,and has become a new tool to deal with the massive data in the era of big data.In this paper,we first introduce the large environment of mass network traffic,including the abnormal network traffic and abnormal detection.Then,some related technologies of large data processing are introduced.Subsequently,the traditional detection method of abnormal network traffic is analyzed,pointing out the shortcoming of the traditional detection method,and then we proposed a new detection algorithm based on the combination of Grubbs anomaly detection method and dynamic K-NearestNeighbor algorithm as the algorithm for real-time detection of abnormal traffic,and the rationality and correctness of the algorithm are verified.And then we introduce the abnormal traffic monitoring system in detail,including Kafka message system,Storm stream processing platform and HBase non-relational database,and the application of the algorithm called KNN-Grubbs anomaly detection method in this Storm system framework.Finally,we analyze the experimental data of the abnormal network traffic detection system.All of the data were derived from more than 2000 enterprise network daily real flow in some province,After the extraction of the field to get a custom composite session data,and then carried out a further calculate of some properties of composite session data to get the host flow real-time table,so the authenticity and mass of data can be guaranteed.In this paper,all the tests are completed on the Storm real-time processing platform.And the experimental results show that the system can accomplish the task of real-time monitoring of abnormal traffic.