Research On Network Anomaly Detection Based On Ensemble Learning

The Internet has become a critical infrastructure that profoundly affects all aspects of people's lives,along with the continuous advancements in information technologies and the explosive growth of various network applications.While the Internet is improving the standard of living and production efficiency,and promoting economic development and social progress,the security problems of network have become increasingly prominent.Various types of network intrusions and attacks emerge one after another,which poses a serious threat to cyberspace security.Intrusion detection is the basic technology and key step against various cyber threats.Different from the traditional signature-based or statistic-based detection methods,the machine learning-based network anomaly detection technology can not only detect known types of attacks,but also detect unknown types of attacks,therefore it has attracted lots of attention in recent years in academia and industry.However,the anomaly detection method obtained by researches is still far from actual deployment at present.In order to solve this problem,we first analyze various existing network anomaly detection methods,especially focuses on analyzing its detection accuracy,false alarm rates,and real-time deficiencies.We propose an ensemble learning-based network anomaly detection technology,and obtained the following research results.Firstly,we survey the machine learning techniques used in network anomaly detection,and analysis from the perspectives of learning supervision mode,the granularity of input data,and the scope of applications.We then point out the shortcomings of existing methods,and analyze the reason for the shortages.We consider that ensemble learning-based detection has great advancement in the detection of network anomalies in the face of big data challenges.The increasingly mature of massively parallel and distributed computing platform technology also provides opportunities for machine learning-based network anomaly detection.Secondly,we propose a novel network anomaly detection approach,HELPAD,which uses ensemble learning method based on Hidden Markov Models(HMMs)to find out malicious network packets.HELPAD focuses on the context of network packets,and its detection object is the network packet sequence.In order to reduce the false alarm rate,an ensemble learning method is used to eliminate the deviation caused by the random initialization of the model parameters.During the training phase,the TCP flags are extracted from normal packets and transformed to sequences of decimal numbers,and then these sequences are used by Baum-Welch algorithm as observations to determine the optimal parameters of HMMs.These HMMs which represent the normal behavior patterns of the network are used as multiple classifiers to decide whether the packets sequence is malicious.Experimental results obtained on DARPA'99 datasets show that the analysis performed by our approach is particularly effective.Thirdly,in the face of the increasingly complex,covert and sophisticated cyberattacks,we propose a network anomaly detection model based on the integrated analysis of transport layer and application layer message.HELPAD is firstly used to detect the transport layer message sequences,and then the positive results of the detection are checked by a byte-level application-layer payload detection to reduce the false alarm rate.We propose an anomaly detection method,M-N-Grams,based on multi-granularity ngram model for application layer payload analysis,which characterizes the normal behavior of network packets at the application layer based on the character context and character sequences.We also propose a probabilistic tree structure model to improve the storage and analysis efficiency of M-N-Grams algorithm.Experiments on the CSIC 2010 dataset and the DARPA 1999 dataset show that M-N-Grams is an effective anomaly detection method for application-layer payloads.Meanwhile,the HELPAD + M-NGrams model can be used without any significant increase in system overhead to reduce false alarm rate by 50% to 70%.Fourthly,concerning the serious threat posed by sophisticated emerging attacks such as APTs,we analyze the semantically rich behavior patterns of network applications and propose the deep network behavior analysis theory and framework for detecting sophisticated emerging attacks,based on the integrated analysis of the message layer,protocol layer,operation layer and traffic layer behaviors.Combined with protocol reverse analysis and big data stream processing technologies,we establish a set of basic theory and technical framework of network behavior patterns,forming a novel application mode for network anomaly detection.Based on the HELPAD and M-N-Grams,we implement an example of network threat detection based on deep behavior analysis on the Storm platform.