Network Anomaly Traffic Detection Technology In Stream Computing Environment

The development of the Internet,the growth of bandwidth and the evolving of the network itself and the network attack makes the abnormal detection algorithms and systems face challenges in accuracy,timeliness and efficiency.At present,the network anomaly detection scheme of big data processing platform is based on the Hadoop data processing platform.They deploy the Hadoop data processing platform on the cluster and calculate the collected traffic data by using various machine learning algorithms without considering the real-time performance of abnormal traffic detection.It is difficult to meet the requirement of real-time detection of abnormal feedback traffic.At the same time,the detection algorithm has the optimized upgrade space and can be further optimized to adapt to the large data distributed processing mode.In addition,the detection model which generated by a machine learning has the problem of validity and can not always adapt to the ever-changing network environment.In view of the above problems,the main contents are listed below:(1)A new selection algorithm for initial clustering based on single point density and distance is proposed.This thesis analyzes the existing core idea and algorithm flow of K-means algorithm based on unsupervised clustering algorithm.Aim at the problem that the algorithm is sensitive to the initial clustering center,a new selection algorithm based on single point density and distance is proposed to reduce the iteration times,prevent it from falling into local optima,and improve the detection rate.(2)A method to adaptively update the detection model is proposed.This thesis proposes a method to adaptively update the detection model,which makes the model no longer restrictive and can change with the change of network traffic characteristics and can adapt to the changing network environment.(3)A Spark-based network traffic anomaly detection system is designed and implemented.The network traffic anomaly detection system is designed and implemented in the flow computing platform Spark.The system is divided into preprocessing module,detection model generation module,abnormal traffic identification module and model update module.The improved abnormal traffic detection algorithm is applied to the generation of the detection model and the method to adaptively update the detection model is applied to the model update module,so that the anomaly detection system can carry on the online real-time detection and improve the efficiency.Finally,the efficiency and accuracy of the system are analyzed experimentally.