Network Anomaly Detection Based On Data Mining Techniques

Network anomaly detection plays an important role in network security. It pro-tects our computer system and network from intrusions and keeps business running steadily. With the increasing development of machine learning and data mining techniques, those approaches are becoming far more popular in the application area of network anomaly detection. Though, those techniques have advantages of au-tomation, high-performance and capability to detect unknown intrusion, they also suffer from a series of challenges, such as high dimensions, label lacking, low efficien-cy and high false positive rate, which indicates that they need furtherly improved.The main work of this theis is the application and adaptation of data mining techniques in network anomaly detection, specific content as followsWe propose an unsupervised feature selection algorithm based on maximal in-formation coefficient (MIC) and clustering due to the high dimensions of network data and label lacking conditions. With MIC, we obtain the optimal mutual infor-mation between two continuous features. In proposed method, discrete and contin-uous features are handled respectively. Simulation experiment are carried out and the results show that proposed method achieves the comparable accuracy with other supervised feature selection methods in network data without label.We propose an unsupervised data mining approach based on density peak(DP) clustering. Though DP clustering algorithm considers cluster centers in a new per-spective and choose cluster centesr in an intuitive way, it has bottleneck in space complexity. We adapt this method using reasonable and unbiased sampling oper-ation, not only reduce the space requirement but also make it faster. Experiments results over network intrusion dataset shows it’s advantage in runtime and accuracy, and other measurements as well.We propose an approache for network traffic anomaly detection in Tcpdump format in order to satisfy the the timeliness requirement of detection. Improved K-medoids clustering is applied to detect outliers points and accelerate the iterative process by choosing approximate centers. To alleviate the high false positive rate, we introduce fp-growth algorithm to mine valuable rules for further outlier detection. Simulation experiment on network traffic indicates that proposed method is effective and efficient.