Network Flow-based Traffic Anomaly Detection

With the development of network technology and the popularization of global Internet, network has been a necessary tool for bearing the weight of information. However, this also results in the menaces from various kinds of security incidents, for example, abnormal traffic brought by multifarious attacks and virus. The traditional intrusion detection system would not be applied to large-scale network environment. Relatively, the network anomaly detection system based on network flow has higher efficient and cost-effective, which can detect traffic anomaly in a higher level. Therefore, based on network flow traffic anomaly detection have an important research value and practical significance. In addition, network traffic anomaly detection should not only detect whether there is abnormal flow, but also to find the reasons leading to abnormal, that is, to identify anomalies in network traffic flow, and accurate positioning abnormal host. This article is based on the above issues, have done a lot of research works as follows:First, this paper sammarizes the emergence, development and collecting methods of network flow, and compares three commonly methods of collecting network flow, and explains the reasons for choice of NetFlow and it's advantages.Second, this paper analyses popular abnormal network flows'characteristics, and presents a cluster-based abnormal flows detection algorithm. It is designed base on Chameleon algorithm, which is a hierarchical clustering using dynamic modeling, and it would distinguish abnormal flows without future knowledge.Third, this paper also designs a NetFlow-based network anomaly detection model, which consists of NetFlow data collection module, NetFlow flow data statistical module, anomaly detection module and alert module. In NetFlow data collection module, we filter single flows with obvious abnormal characteristics, and then save the flow data to database. In flow data statistical module, flow would be aggregated according to definite rule, and save to database.Anomaly detection module consisted of network traffic anomaly detection submodule and abnormal flow detection submodule. Firstly, we use multi-feature similarity method to detect network anomaly. If similarity between forward and after time is larger than standard threshold, we consider network is normal, and then update standard threshold using weighted average method. Contrarily, we think network traffic is abnormal, and then abnormal flow detection submodule is activated. In the submodule, we construct a k-nearest-neighbor graph from original network flows dataset firstly. Secondly, we partition the graph into several subclusters, and then discover the genuine cluster in the data set using the dynamic modeling framework to hierarchically merge these subclusters. Thirdly, we filter the normal flow clusters that had higher relative closeness, and the flows of other clusters are abnormal traffic. At last, we update the standard threshold according to detecting result.Finally, we tested the network anomaly detection system. Through the test of DARPA98, we found the abnormal flow algorithm can detect abnorma flows accurately. Through simulation experiments show that the prototype system can discover network anomaly availably and detect abnormal flows.