Anomaly Detection Of Network Traffic Based On Unsupervised Learning

With the rapid popularization of the Internet,network security incidents emerge in endlessly,data security,privacy protection and other issues have been paid more and more attention.As a result,network anomaly detection has become a hot research topic.Many researchers begin to use machine learning to overcome the problems in the field of anomaly detection,and have achieved a lot of experimental results.Traditional machine learning usually needs to manually select features for model training.If the features are not selected properly,the accuracy of model detection will be low,and the model will fall into local optimization,resulting in weak generalization ability.Unsupervised deep learning does not need to manually select features,it can solve the problem of dimension disaster,and mine potential rules from unlabeled data.It also has good detection ability for unknown abnormal traffic.Therefore,network traffic anomaly detection based on unsupervised learning has become one of the core of the research.This paper studies and implements a system of network abnormal traffic detection,and designs a distributed cluster architecture for deployment.The purpose is to detect the network abnormal traffic in real-time and efficiently in the real network environment.This system mainly includes three modules: the first is the real-time network packet collection module,the second is the real-time network abnormal traffic detection module,and the last is the visual management module.In order to ensure the real-time data packet collection,this paper designed a high-speed network traffic collection architecture based on DPDK(Data Plane Development Kit)to capture network traffic in real-time.Aiming at the problems of large amount of network traffic data,high feature dimension and strong dependence of supervised learning on data labels,this paper designs an abnormal traffic detection algorithm based on unsupervised learning.By improving DAGMM(Deep Auto Encoding Gaussian Mixture Model),the unsupervised anomaly detection algorithm based on CAE-GMM(Contractive Auto-Encoders Gaussian Mixture Model)was designed,and the model hyperparameter tuning was carried out.The CAE-GMM model in this paper not only combines the feature reduction of CAE and the density estimation process of GMM to carry out end-to-end joint training,but also the CAE network can effectively remove noise interference while carrying out feature reduction.Compared with the original DAGMM,the CAE network can reduce the overfitting problem and improve the generalization ability of the model.For abnormal discrimination,the sample energy value is calculated first,and the abnormal flow is defined when the sample energy value exceeds the adaptive threshold value.Finally,this paper designs and builds a distributed cluster architecture to ensure the high performance and high availability of the system.In this paper,the abnormal traffic detection model was tested on two data sets,KDD99 and CICIDS2017,respectively.The experimental results show that the accuracy of the abnormal traffic detection model in this paper reaches 96%,which is about 3% higher than the original DAGMM model.The abnormal flow detection system in this paper has been deployed in the actual application scenarios of relevant medical institutions for trial operation.During the operation,the system is stable and the abnormal alarm is timely and accurate,which provides a certain guarantee for the data security of medical institutions.