Multi-feature Similarity-based Large-scale Network Anomaly Detection

With the development of network technology and the Internet, the growing size of the global Internet and the increase in the number of Internet users, Large-scale Internet attacks and the spread of the virus are more and more frequent. How to protect the security of networks and information systems has become a topic of great concern to people. The traditional intrusion detection system for abnormalities should not be applied to large-scale network environment. For the network flow analysis is a more natural way. Using dynamic changes of flow behavior as a study object can study characteristics of the network itself at higher granularity, which is an important field of research.The characteristic of large-scale network data flow is continuous, fast and large-scale, since there is no real solution to the detection and analysis of network data flow by flow by data mining techniques. This paper presents the network characteristic attributes (NCA) to describe the state of large-scale network; since the combination of network traffic with similar characteristics, we put forward that using NetFlow technology to carry through flow collection, using multi-feature similarity method to carry through anomaly detection. Our experiment also demonstrated the effectiveness of the method which has theoretical and practical value.A Large-scale Network Anomaly Detection System (LNAD) consists of NetFlow data acquisition module, NetFlow flow data preprocessing module, training data module and anomaly detection module. In NetFlow data acquisition module, we set up distributed warning agents for receiving NetFlow information flow to meet the demands of large-scale networks. In flow data preprocessing module, we aggregate network data by the method of high-frequency statistic according to the demand of the network attributes. Then the data will be deposited into the database. In training data module, according to the cyclical characteristic of network traffic, we calculate the standard network feature matrix (SNFM) and the standard similarity threshold (SST) on different time slots based on the training data. This will be better for Anomaly Detection. In anomaly detection module, we unify the dimension of various network attributes, compare with the threshold standard for coarse anomaly detection through the improved multi-feature similarity algorithm. When we find abnormity in Coarse Detection, according to changes of similarity when abnormal, we carry out a further testing by Exact Detection. When no abnormality found in the final twice testing, real-time data is also used as a new training data to meet the ever-changing features of network traffic.Finally, we conducted a test of the system network. Through simulation of network attack, we found the prototype system can discover network abnormality and achieved the goal of real-time network traffic anomaly detection. We also compared the results of the algorithm and the improved algorithm for the influence on network anomaly detection sensitivity.