Research On Network Traffic Anomaly Detection Over Sliding Windows Under Hadoop

With the rapid development of information technology,the network has been closely related to people's production and life.Network traffic is growing at a tremendous speed,at the same time,cyber security have become increasingly prominent.With the arrival of the era of big data,in the field of network traffic safety,traditional network traffic anomaly detection technology is facing the following major challenges: the accuracy of anomaly detection is not high enough,response rate is not fast enough,system storage and processing capacity is insufficient,etc.With the method of hierarchical clustering and sliding window,this paper proposes an anomaly detection algorithm based on Canopy algorithm and weighted K-means algorithm,and then explores the parallel design and implementation of the algorithm in Hadoop platform.It can detect the mass of network traffic data in real time,and also has the characteristics of fast response and high accuracy.The work of this thesis are:1.“Research on network traffic anomaly detection method based on Canopy and weighted K-means”.Based on the Canopy algorithm,this paper proposes a hierarchical clustering algorithm for high dimensional data stream clustering based on sliding windows-CHSWStream(Canopy High Sliding Window Stream).The online component uses PCA for feature dimension reduction of high dimensional data,uses Canopy algorithm to achieve the initial clustering,then extracts summary information by using exponential histogram technique,and updates expired data via sliding windows.The offline component uses the mature clustering algorithm-weighted Kmeans algorithm,and finally realizes the unification of detection rate and detection accuracy.2.“Research on parallel network traffic anomaly detection method based on hierarchical clustering in Hadoop”.According to the CHSWStream algorithm designed in the last section,this paper studies the parallel design and implementation of the algorithm under MapReduce architecture.Experimental results show that the improved algorithm can improve the running speed while running under Hadoop.