Key Technologies Research On Traffic Anomaly Detection And Optimization For Large-scale Networks

With the continuous development of network science and the explosive increase ofall kinds of network applications, the Internet has become an important infrastructurewhich deeply affects politics, economy, military affairs and culture. Meanwhile,problems of network security have become more and more serious. Flash crowd andnetwork based attacks, such as DDOS, worm burst, large-scale port scan, becomeincreasingly frequent. The Internet confronts with serious security challenges. As aneffective means of network protection which can detect unkown attacks, traffic anomalydetection is increasingly accepted by scholars and industrial sectors. However, with thecontinued growth of network bandwidth and the sustained of the gamebetween attackers and defenses, the network which is ongoing evolution is dynamicprocess and the network attacks are evolving also, resulting in that anomaly detectionsystems are challenged in detection accuracy, operating efficiency, safety and ease ofuse. For example, generalization of anomaly detection systems is poor; the dynamicnature of network traffic which is abrupt and drifting led to detection accuracy is poor;feature-based anomaly detection systems can not meet the rapid growth of networkbandwidth; some kinds of anomaly detection systems vulnerable to attacks; the trainingdata are difficult to obtain. Therefore, proposed high precision and high detectionefficiency of anomaly detection systems and optimized detection algorithms is greatsignificance for large-scale communication networks which is the focus of the globalacademic and industrial of network security.Firstly, a classification based anomaly detection system which usesmulti-dimensional time series of entropy for backbone networks is proposed. Theexisting detection systems based on entropy has been widely adopted, but time andspace complexity of calculation entropy is high. The correlation of time series ofentropy at each windows and the correlation between multi-dimensional time series ofentropy are overlooked by researchers. The study found that attack packets are usuallygathered at a fixed eigenvalue on certain dimensions, in the other dimension is evenlyspread in the value space, and network traffic itself in all dimensions showingpower-law distribution, it is easy to divided into large and small flows which will beused to improve the estimating algorhtim of entropy. The proposed approach utilizesentropy to measure the distribution of traffic over some traffic dimensions (features) anda efficient algorithm is introduced to estimate entropy with low computational and spacecomplexity; the values of entropy over all dimensions are collected to form a detectionvector in every sliding window, and then all detection vectors are classified into twogroups: abnormal vectors and normal vectors via one class support vector machine; Inorder to achieve the goal of high detection rate and reduce the false positive rate, a multi-windows correlation algorithm is presented to calculate comprehensive anomalyscores when observing a sequence of windows. Some real-world traces were used tovalidate and evaluate the efficiency and accuracy of this system through threeexperiments.Secondly, a novel anomaly detection system which can filter malicious packetsbased on Filter-ary-Sketch is presented. Anomaly detection systems are usually onlysent a warning when an anomaly occurs, but some methods cannot be proposed toinhibit the anomaly by network managers with out the help of detailed informationabout the anomaly. Meanwhile, the feature-based detection systems while having highdetection accuracy have some security flaws, an attacker can construct a specialnetwork attack to bypass the detection system. The study found that network traffic canbe seen as data flow, the hash summary data structure built on to get traffic in eachdimension. It records the traffic in Filter-ary-Sketch and detects anomalies over it.When an anomaly is detected, the anomalous dimension is notices and maliciousbuckets are identified. Finally, malicious packets are blocked using the packets filteralgorithm. Using some traffic traces from backbone networks, we demonstrate that oursystem can detect anomalies with high accuracy, low computation and memory costs,and block the packets that are responsible for anomalies.Thirdly, an anomaly detection system based on correlation analysis to optimize thedetection accuracy is prososed. With the continuous development of network security, avariety of traffic anomaly detection systems (TADS) have been proposed. However,temporal and spatial correlation of them is not understood very well. In this paper, thespatial-temporal correlation of some metrics is analyzed firstly; the alarms collectedfrom multiply windows and various types of anomaly detection systems are correlatedwith each other and this correlation is used to reduce false positive rate. We use supportvector regression to predict the value of anomaly scores, and the multi-windowscorrelation algorithm is adopted to cumulate multi-steps of the margin. Usingmulti-metrics correlation algorithm to fuse results of the detection systems, the finaljudgments is achieved. Finally, using some real and synthetic traffic trace frombackbone networks in the core of Internet, we demonstrate that our system can improvethe detection rate and control the false positive rate significantly.Fourthly, the problem about how to extract and train the classifier in trafficanomaly detection systems is analysed and some datastructs and algorithms areproposed. With the continuous development of network security, various types of trafficanomaly detection systems have been proposed and the system based on classification isone of the most important classes. However, because the network environment isdiversity and dynamic changing, the traffic anomaly detection systems confront withhigh false positive rate when it deployed who can detect every anomaly in the trainingdataset exactly. It is difficult to extract precision classifier and classifier should adapt with time and environment. In order to overcome these problems mentioned above,algorithms and data struts about how to extract and train the classifier were proposed.The network traffic was projected to different Hash histograms over differentdimensions, and these histograms, through a support vector data description machine,were classified into two groups, abnormal and normal; secondly, in order to improve theaccuracy of detection and reduce training time, the classifier was trained continuously todescribe the real-time and dynamic network traffic exactly. Third, the multi-windowscorrelation algorithm was adopted to calculate a comprehensive anomaly score througha sequence of windows and weed out obvious outliers in the new training set. Usingsome real and synthetic traffic traces from backbone networks in the Internet, wedemonstrate that our system can detect anomalies with higher accuracy, lowercomputation and memory costs than other systems which were widely used in anomalydetection.In summary, we focus on detection traffic anomalious for large-scalecommunications network with high detection accuracy, efficiency, safety and ease ofuse and the problems about how to optimize the detection systems. These works haveacademic and practical value for advancing the theory and practicability of the aboveresearch.