Research On Real-Time Network Traffic Anomaly Detection Algorithm And System Realization

In recent years,with the rapid development of the Internet,various network applications are emerging,and the number of Internet users in China and even in the world is increasing day by day.With the popularization of the Internet,people are getting closer to the Internet in daily life and the data of network traffic generated by the Internet are also getting bigger and bigger.But at the same time,the hidden dangers and threats posed by cybersecurity issues are also growing.Network security incidents are often accompanied by abnormal network traffic fluctuations,so fast and effective detection of network traffic anomalies in the prevention and management of network security issues of great significance.This thesis first studies the network traffic and proposes an anomaly detection algorithm based on STL decomposition and ESD verification for the periodicity of network traffic.The data is used to test the anomaly detection algorithm for parameter selection and tuning.Then,on the basis of the batch anomaly detection algorithm,the flow modification is made and a real-time network traffic anomaly detection algorithm is designed.Then,based on the flow anomaly detection algorithm,a real-time anomaly detection system is designed and implemented using big data framework such as Kafka and Spark Streaming.The composition of the real-time anomaly detection system is introduced in detail,including the input part by Kafka,detection part by Spark Streaming and the output part of the java web.All the experimental data in this thesis come from the real network traffic of a province in north China from August 1,2015 to August 31,2015.Experiments show that the anomaly detection algorithm proposed in this thesis can really find the anomalies in the network traffic,and the real-time anomaly detection system can correctly accomplish the task of real-time monitoring of anomalous traffic.