Multidimensional Clustering Based Anomaly Detection

Network anomaly detection which is a very important issue in network management has been extensively studied in recent years. Although people in the field made a number of advanced works, the accuracy of automatic classification of network traffic to detect and identify abnormal network traffic is still a very challenging problem. There are three basic methods of anomaly detection, supervised anomaly detection, semi-supervised anomaly detection, unsupervised anomaly detection. Although supervised anomaly detection method is able to identify abnormal network traffic by establishing an accurate normal traffic model, the need for manually labeling data to get enough training samples will cause a lot of waste of human resources. Therefore, unsupervised anomaly detection method is proposed. Clustering is an unsupervised anomaly detection method in the field of intrusion detection; it can extract useful information from the samples with no classification marks. Clustering method does not require a priori knowledge and can discover unknown attacks. But due to the diversity of attacks, the traditional clustering algorithms can’t work very well in mining network traffic anomalies. This paper proposes a multi-dimensional cluster analysis based anomaly detection method, by two stages to achieve anomaly detection. The first phase, through multidimensional clustering algorithms, network traffic is automatically mined into different multidimensional clusters. The second phase calculates the degree of multidimensional clusters to achieve anomaly detection. The basic idea of cluster analysis is introduced and several typical clustering algorithms are analyzed firstly. Then a detailed description of one-dimensional and multi-dimensional clustering algorithm over data streams is given. But the network is characterized with massive data streams, the original multi-dimensional clustering algorithm is not feasible in practice. A number of optimizations are introduced to improve the efficiency of clustering. Then a method used to calculate the abnormal degree of clusters is presented to better identify abnormal traffic generated from network attacks. On the basis of the theory, a multi-dimensional cluster analysis based anomaly detection system is designed and implemented and a detailed description of the system architecture and main modules is given. Finally, different data sets of network attacks is used in the system test, the experimental results show that the anomaly detection system can effectively detect scanning, DDOS, worms and other types of network attacks.