Research On Network Traffic Anomaly Detection Based On Distributed Computing Technology

The rapid development of the Internet technology not only give us very convenient service,but also bring new challenges.Firstly,the traditional network threats are still severe,and many new threats continue to come out,which makes difficulties for network anomaly detection.Secondly,many interesting and useful software applications attract people to generate huge amounts of traffic volume by computers and mobile devices.Thirdly,commercial IDS can detect the known attacks in high speed,but it cannot deal with the unknown anomalies and cannot store the huge amounts of traffic.Fourthly,although traffic anomaly detection technology can find the unknown traffic anomalies,its detection capability is not very high.To address the problems,an effective way is,continuously improving the traffic anomaly detection technology and storing and processing capability by big data technology.In this thesis,the network traffic anomaly detection technology is researched by collecting the flow-level flow of the edge or core routers.The main contents and contributions are as follows:1.We propose a network traffic anomaly detection model based on distributed computing framework,which includes a distributed computing model of new metrics for anomaly detection and an anomaly detection model based on the metrics.2.We propose both DTE entropy and a traffic anomaly detection approach based on DTE entropy.DTE entropy is an improvement of Tsallis entropy for network traffic anomaly detection.it uses Tsallis entropies of two most efficient q values to form Tsallis entropy pairs to improving detection performance.The experiments show that the approach based on DTE entropy greatly improves the detection efficiency compared with the Tsallis entropy approach.3.We propose APE entropy model to deal with the problem of traditional entropies,which isn't effective enough for the fluctuation of the flow number,a small scale of anomaly flows and the situation of entropies cancelling out.And we also present a detection algorithm based on the APE entropy change rate.Furthermore,we validate the APE entropy by APSE,an instance of APE entropy,which proves that APE entropy has higher detection capability than tradition entropy for traffic anomaly detection.4.In order to achieve fast,detailed and efficient traffic anomaly detection and classification,we propose CEFF and implement it both on Spark and Spark streaming platforms to realize the online and offline detection and classification.5.In order to meet the needs of experiment and actual detection,we design a network traffic anomaly detection system based on big data technology.The system not only achieves online and offline detection and classification based on Hadoop,Spark and Spark streaming,but also supports data querying base on Hive.