Research On Anomaly Detection Technology In High-speed Complex Network Environment

With the development of information technology and the improvement on socialinformationlizing, Internet becomes more and more important in the life of the human being. Thenumerous protocols and applications and the complicated behavior of billons of network userstogether form a super-scale network system. So, the robustness is not only the requirement of thedevelopment of Internet itself and it is also the fundamental issue of the development of humansociety. In recent years, cloud computing has boomed up and the change of informationapplication to service and the transforming of service to the cloud end are accelerated.Based on the research work that the author of this dissertation has done in the project of“High-dependable Network Business Management and Conformity of System”(2009AA01A346)and nation’s science and technology shore up project “Large Scale Access Converge Router, thisdissertation deeply investigates the anomaly detection technology in the high speed complexnetwork environment. And the main contributions of this dissertation are as following.1. It provides a systematic survey of the current stage of the research on anomaly detection.And the requirement of anomaly detection technology in high speed complex networkenvironment and the methods proposed are deeply compared. And the anomaly detection methodbased on hybrid granularity is proposed in this dissertation.2. The Flow Size Adaptive Sampling（FSAS）for abnormal flow length distribution isprovided on the basis of the abnormal traffic detection. After the analysis on the performance ofFSAS and simulation, the results show that the FSAS not only effectively reduce the vastamounts of data but also maintain the characteristics of suspected abnormal traffic.3. It provides the traffic description based on the symbol sequence and conduct deep studyon its features in theory. The method transforms the complex mass flow data into the sample dataof limited set from the view of mutual information in the information theory. By defining thesymbol CMI (Continuance Mutual Information), the paper analyze the unique statisticalproperties of symbol sequence, define a new metric index to describe the unique statisticalproperties of abnormal traffic, and assess its feasibility for the abnormal traffic detection.4. Considering the existence of long range mutual information and its influence on trafficmodeling, a new symbol sequence oriented traffic model is proposed. The new traffic model isnamed as TSTM, and it has several appealing merits. Firstly, TSTM can capture the CMI ofInternet traffic precisely. Secondly, TSTM is a structural traffic model and it has a simplestructure. More appealing one is that there are only seven parameters in this model, and all theparameters has its physical meaning and can be estimated from the real traffic data. Experimentalresults with real traffic traces demonstrate the validation of the traffic model and anomalydetection method.5. It provides URCA (Unsupervised Root Cause Analysis) which includes two levels basedon the traffic symbolization and TSTM model. Experimental results with real traffic tracesdemonstrate the validation of the traffic model and anomaly detection method.6. The classification and identification system of abnormal traffic which could embed ACR is designed by combining with the subject requirements. The detailed results and practicalapplications could validate the feasibility of such classification and recognition system.