Research On Network Anomaly Detection Based On Machine Learning

Information and communication technologies has a great impact on the social well-being,economic growth and national security of today's world.However,it has also been used by some people with bad intentions to endanger cybersecurity and seek illegal inter-ests.Prohibiting these harmful network activities is one of the international priorities and important research areas,and its primary task is to identify the attack activity in network traffic,that is,use the intrusion detection system for network anomaly detection.Nowa-days,there are many problems that slower the development of network anomaly detection technology.This thesis will focus on two of them.First,the data set used for network anomaly detection has high dimensional and non-linear characteristics,which limits the effect of traditional machine learning algorithms.Second,due to privacy reasons,there is a lack of labeled public dataset.This thesis proposes and implements three machine learning-based methods for intrusion detection systems to solve these two problems.The first is to use the RFE(Recursive Feature Elimination)based on CART(Classification and Regression Tree)decision tree to achieve feature extraction.Using RFE for feature ex-traction can reduce redundancy and invalid features in the data set,thereby improving the effectiveness of traditional machine learning algorithms and significantly reducing time consumption.By referring to the features retained after feature extraction,it is possible to reduce the required features when collecting data from traffic,which helps to reduce the burden on the host and avoids the trouble that may be caused by collecting privacy data from users.The second is to implement an intrusion detection system using a neural network based on single LSTM(Long Short Term Memory Networks),which confirms the fact that the deep learning method performs well on high-dimensional nonlinear network anomaly detection data.In the scenario where the collected network traffic data has high dimen-sions and is inconvenient for feature extraction,or when the training time is suff-icient,the neural network can be used to obtain a very good result.It is superior to feature extraction in the pursuit of accuracy and rich host computing resources.The third is an unsupervised intrusion detection system based on VAE(Variational Autoencoder),which obtains a high accuracy on the network anomaly detection dataset,and verifies the availability of the unsupervised learning algorithm in network anomaly detection.Since the unsupervised learning algorithm does not need the labeled data,the method can be trained using unlabeled network traffic,which helps to alleviate the prob-lem that the network anomaly detection has insufficient labeled datasets.