Research On Analysis And Detection Of Multi-layer Abnormal Behavior For Network Data Stream

Network anomaly detection is one of the supporting network security technologies. The traditional anomaly detection technology is increasingly unsuited to the complex communication network environment. Network anomaly detection technique based on behavior analysis gradually become the hot topic in the field of anomaly detection due to the advantages of mining intrinsic information and mutual relations as well as flexibility, adaptability, etc. In recent years, although the acchievements of research in this area is remarkable, but there are still some problems. For example, most of the current researches focus on a single behavior layer. They are difficult to completely reveal the cause and nature of the exception and can not provide support for abnormal blocking and handling fully.To solve this problem, the paper studied abnormal behavior detection technology based on network data stream in three layers:traffic behavior layer, protocol behavior layer and user behavior layer and achieved the following results:1. In traffic behavior layer, the paper proposed an abnormal traffic behavior detection method based on clustering model evaluation. Aiming to the problem that feature extraction is difficult and the detection capability of existing unsupervised technology for scale anomaly is not high, unsupervised clustering technique is used and the paper proposed a weighted feature selection algorithm for clustering which combined information entropy with neighborhood analysis technology for feature evaluation and selection and improved efficiency of feature reduction and clustering performance. Then the concept of clustering rules and cluster model are difined. The rules were improved by clustering the training phase and cluster model is used in K-means clustering algorithm at real-time detection stage to calculate deviation degree of real data stream. If the deviation degree exceeds a threshold value, it is determined that appear scale anomaly appears and the entire flow will be output, otherwise, outliers or subclass clusters will be output. This method not only retains the handling capabilities for general exception, but also to be more suitable for scale anomaly detection. Experimental results show that the proposed method which more suited for scale anomaly detection ensured the detection efficiency and decreased the false positive rate and false negative rate.2. In protocol behavior layer, this paper proposed an abnormal protocol behavior detection method based on conditional random fields (CRF). Aiming to the problem that existing protocol anomaly detection method based on hidden Markov model is too strict in dependence assumption and lack of useage of context features, this paper used protocol keywords, time interval and their frequency characteristics to describe protocol message and modeled protocol packets series based on CRF. On this basis, abnormal behavior is determined by calculating the joint occurance probability of observed packets sequence in this model. This method combines state characteristics and frequency characteristics of protocol packets, while taking CRF’s advantage of more complete and accurate description of the protocol behavior without independence assumption. Experimental results show that compared to the traditional method it improves the ability to detect abnormal protocol behavior.3. In user behavior layer, this paper presented an abnormal user behavior detection method based on conversation associated analysis and behavior pattern drift detection method based on relation entropy and J values. Aiming to the problem that complete description of user behavior and establishment of behavior patterns are difficult in traditional abnormal user behavior detection technology, the paper introduced the session occurrence frequency to improve and define the fuzzy temporal association pattern and modeled user session sequence, then based on pattern matching to detect the occurrence of abnormal user. Relation entropy and J values are introduced to quantify session relationship, and then put forward pattern drift detection algorithm based on the hypothesis testing. Experimental results show that the method is accurate and feasible.The above work provides for analysis and detection of abnormal behavior in three layers, and coveres the shortage that traditional research was limited in single layer. In addition, it improves the efficiency and effectiveness, while providing further support for accurate early warning and rapid handling of anomaly.