Research And Implementation Of Network Anomaly Detection System Based On Machine Learning

The rapid development of information technology and Internet has brought great convenience to all walks of life.However,with various network security intrusions and other anomalies emerging one after another network security is becoming a more and more prominent issue,.Traditional technology can no longer satisfy the more and more comprehensive network environments of nowadays,so the detection of network anomalies in a more efficient and reliable way has become a research hotspot in the field of network security.In the current research,there are anomalies detection technologies based on a variety of ideas,of which,machine learning is the current and future research trend.At present stage,machine learning technology in network anomalies detection has yet achieved good results in many scenarios,but there are still shortcomings such as low universality,low accuracy and strong data dependence.In view of this situation,this paper proposes several new ideas for network anomalies detection technology,designs and implements a set of network anomaly detection system based on machine learning.The first one is a variational auto-encoder(VAE)network anomalies detection algorithm model based on gaussian mixture distribution.This model is a combination of deep learning and statistics,it is an unsupervised learning method.The algorithm model will only be trained for the normal data set in the sample data,and the anomaly will be detected by the reconstruction error trained by the variational auto-encoder(VAE)neural network.It can effectively solve the problem of missing data set of labeled samples and insensitivity to unknown abnormal behaviors in network anomalies detection.The second is using the text processing technology to propose a set of original log feature analysis model based on natural language processing.In this model,n-gram and TF-IDF technologies are used in the segmentation of the original logs in the network as text content,and finally the feature vectors required by the machine learning algorithm are obtained.The second is to use the text processing technology in natural language processing to propose a set of original log feature analysis model.In this model,n-gram and TF-IDF technologies are used to segment the original logs in the network as text content,and finally the feature vectors required by the machine learning algorithm are obtained.This model abandons the multifarious steps of manual feature extraction in traditional machine learning,and can play a good detection effect with appropriate machine learning classification algorithm.The third is the program design idea based on microservices.A network anomalies detection system is designed and implemented based on machine learning.Each function of the system is corresponded with a microservice,and data transmission and asynchronous communication are realized between each service through message queues.It is fabricated with delicate visual interface,good versatility and fine expandability.