Unsupervised Network Anomaly Detection Based On Abnormality Weights And Subspace Clustering

With the rapid development of information technology and Internet technology, the information resources we gain from the internet becomes much more richer, and convenient communication has tremendously reduced the distances between people, which, meanwhile, is bringing big threats to the computer and network security, therefore the importance of information network security issues has been overwhelmingly seen. Instant detection of attacks or anomalies of networks has become the vital issue of field of network security. Most traditional anomalies detection models are highly dependent on labeled datasets for training, which can not be obtained without huge costs. Moreover those models tend to turn blind to variant attacks. Data mining is a very common data processing technique, which can be utilized to extract potential fact-conforming rules and knowledge. Clustering technique of data mining is a great unsupervised learning method. It builds models based on unlabeled data to discover known and unknown anomalous data, therefore unsupervised clustering has been mostly combined with network anomalies detection.Based on the research background mentioned above, this paper analyzes the real network environment and selects features based on entropy notion to reduce the complexity of original network meta data. Meanwhile this paper proposes a novel unsupervised network anomaly detection system based on clustering and a new abnormality measure based on local density and global distance. It computes the final anomalies after gaining the abnormality of every flow from every single subspace, which avoids not only the necessity of finishing clustering therefore reducing the complexity, but also the dependence on labeled dataset used by traditional network anomaly detection. On the other hand, this paper also proposes a novel abnormality measure based on distance computing, based on which a new unsupervised anomaly detection model is propsed. Both the models largely improve the accuracy and recall rate of network anomaly detection, as well as reducing the detection time expense. In the end, we have conducted several experiments both real 360 network data flow and synthesized KDD Cup99 data, results manifesting outstanding performance of detection model.