Streaming Technology-based Distributed Network Anomaly Detection

In recent years,network has been experienced a fast development.The dominant traffic has transferred from traditional application like web,mail,ftp and etc to the traffic dominated by p2p,and the traffic volume increased a lot.These kinds of change make it harder for the network management,especially to handle the fast traffic and the application adopted dynamic ports.To identify anomaly traffic is always a challenging question faced by network operators. The anomaly traffic could be caused by flash crowd,resource abusing like DoS,worm virus and misconfiguration.As we know,to monitor network state in real-time is crucial for its availability,but it's hard to achieve this if only makes use of SNMP data and the traffic volume is so large that it is almost impossible for real-time monitoring.So,this paper proposes a method based on flow level data.Flow data can be exported by hardware such as Netflow,Netstream,cFlow,and then sent to flow collector in UDP packet.Distributed structure can not only balance load and solve asymmetric route.The data used is flow;the flow contains source,destination IP;source,destination port,traffic volume and transport protocol and etc.After flow collector parsed flow,the useful information then will transfer to the application layer identification module.As mentioned before,P2P has taken up over 50%of traffic volume,but P2P is hard to identify because the feature of proprietary protocol and dynamic port,so it is very important and very hard to identify it.If we can successfully classify traffic into different category then we can make use of this information to provide support for anomaly detection.Traditional anomaly traffic detection method mainly by build a reliable boundary,if the traffic exceeds a pre-designed threshold will cause an alarm,it is easy to implement but has too much false positive.An alternative method is wavelet transform,it is based the observation that where the anomaly happens,where the frequency will change obviously.This paper both adopts time series analysis and frequency analysis,this method takes advantage of them and bases on categorized traffic,the Holt-Winter forecasting analysis take account the periodic influence and self-correlate analysis can handle the traffic in a period. At last,this paper will give the result of this detection design,and the experiment data adopted is Darpa 1999 provided by MIT Lincon laboratory.The experiment result proves that this method is efficient and available.