Research On Feature Statistical Analysis Based Traffic Anomaly Detection

Network anomalies are usually accompanied with the change of traffic features. Therefore,traffic anomaly detection based on feature statistical analysis is an important research topic in thefield of intrusion detection. There are two notoriously important problems preventing it fromprevailing abroad. Firstly, the existing detection methods are limited to simple combination ofsame weight for multidimensional features which causes low true positive rate and high falsepositive rate. Secondly, the anomaly classification methods treat the overall traffic as processingobject which causes low efficiency and true positive rate. Supported by the requirement ofresearch task of the “High Creditability Network Traffic Management and Control System”project of the National High-Tech Research and Development Program of China (863Program),this dissertation analyzes and solves the problem of existing technology achieving low truepositive rates in anomaly detection and classification, and proposes algorithm, system structureand implementation scheme based on multidimensional features. This dissertation has researchedthe following main contents:1. A model named semi-supervised combination (SMC) is proposed, and a traffic anomalydetection algorithm based on SMC model (SMC-FCM) is designed.Considering low true positive rate caused by simple combination of multidimensionalfeatures, this dissertation presents a semi-supervised combination model (SMC). The modelmerges the detection results of base detectors for multidimensional features, and minimizes theinformation loss by solving nonlinear optimization problem. Also semi-supervised learningmethod exploits labeled data to improve the precision of the model. Under the guidance of themodel, this dissertation designs a traffic anomaly detection algorithm based on SMC model(SMC-FCM). The experimental results show that the algorithm based SMC improves theaccuracy by10%to20%over the base detectors and true positive rate reache s97%with falsepositive rate of5%.2. A novel method named Entropy of Features Based Anomaly Traffic Identification (EFATI)is proposed.Considering low true positive rate caused by identification algorithms which utilize normalfeature model, this dissertation presents a novel method named Entropy of Features BasedAnomaly Traffic Identification (EFATI). The algorithm describes traffc features with dynamicalmodifying model, and the distributions of feature are depicted by entropy. The proposed trafficpartition reduction algorithm iteratively reduces the traffic to anomaly subsets by discardingfows that seem normal. The experimental results show that true positive rate of EFATI reaches97%with false positive rate of5%., and the average precision of identification with many typesof anomalies reaches82%.3. A novel method named Traffic Anomaly Classification Based on Hierarchical Clustering(TAC-HC) is proposed.Considering the existing methods of anomaly classification with low efficiency and true positive rate, a novel model is proposed. The model selects traffic features to build attributevector which are available for distinguishing the type of anomalies. It makes use of geneticalgorithm to optimize the parameter. Under the guidance of the model, a method named TrafficAnomaly Classification Based on Hierarchical Clustering (TAC-HC) is proposed. It classifiesnew anomalies with the number of clusters unknown. The experimental results show that truepositive rate of TAC-HC reaches96%with false positive rate of5%. Also the classificationaccuracy of anomalies with low traffic volumes such as network scan can reach95.3%.4. The implementation scheme of the detection system is designed (FS-ADS).We design an implementation scheme of an anomaly detection system used in HighCreditability Network Traffic Management and Control System making demands on accurate ofdetection and classification. We split it into five sub-modules which are anomaly detectionmodule, identification module, classification module and so on. Anomaly proportion test, timeinterval test and real time test are taken to test the performance of the system designed. Theresults show that the system has the best detection, identification and classification performanceunder the given condition, which are anomaly proportion lower than30%and the time intervalof5minutes. And the system only consumes less than200seconds to process one million traffictraces.