Research On Network Anomaly Detection Technology Based On Adaptive Flow Sampling Measurement

Network anomaly detection, which establishes the normal network traffic behavior modelto detect the abnormal behavior of the network, is an important means of Intrusion Detection. Inrecent years, with the continued growth of the number of Internet users and the rapiddeployment of new network applications, threat of attack against the network traffic hasbecome increasingly serious, distributed denial of service attack (DDoS), botnet and wormattacks etc. occur frequently, they have caused great harm to the normal operation of thenetwork. How to timely perception and fast processing of network anomaly behavior in thehigh-speed network environment, it has a very important significance for ensuring the effectiveoperation of the network and raising the robustness of service providing ability.Combined with the fundamental technique research task of identifying abnormal eventsaccurately in the“Research on Reconfigurable Information Communication Basic NetworkSystem” project belonging to the National Priority Basic Research and Development Programof China(973Program), this dissertation primarily discussed how to better detect networktraffic anomaly based on measurement in high-speed backbone link. Considering the trafficprediction model can make reasonable accurate inference for the dynamic trend of networktraffic behavior on different time scales, the paper achieves accurate perception of abnormalbehavior through the combination of traffic prediction method and machine learning method.First of all, this paper realizes the coarse detection of network traffic abnormally through themultidimensional time scales prediction. Then, to prevent the emergence of false positive, ituses machine learning method to carry out fine detection for normal traffic judging from thecoarse detection module. The main research contents of this paper are outlined as follows:1. Aiming at the deficiencies of the existing common sampling methods, a feature-awareadaptive flow sampling (AFS) algorithm is proposed. The algorithm can correct the samplingprobability to minimize the distortion of the traffic feature distribution through the combinationof adaptive sampling method and late sampling technique. In the algorithm, the fows areselected according to the size of their moments, thus it can ignore the redundant flow and focuson the small flows which play an important role in anomaly detection. Compared with therandom flow sampling algorithm, AFS algorithm reduces the loss of information caused by thesampling process, and the anomaly detection capabilities of the system has been improved.2. Statistical characteristic of network traffic data in the fractional Fourier transform (FrFT)domain are analyzed, which indicates the self-similarity feature. Further, Hurst parameterestimation methods based on modified ensemble empirical mode decomposition-detrendedfluctuation analysis (MEEMD-DFA) and adaptive estimator with weighted least square regression (WLSR) are presented, which are aimed at the displaying network traffic in―time‖or―frequency‖domain of FrFT domain separately. Experimental results demonstrate that theMEEMD-DFA method has more accurate estimate precision but higher computationalcomplexity than existing common methods. While the overall robustness of adaptive estimatoris satisfactory over the other six methods in simulation, and that it has lower computationalcomplexity. Thus, it can be used as a real-time online Hurst parameter estimator for traffic data.3. Aiming at the self-similarity of network traffic on large-time scale, a traffic forecastingmodel based on modified ensemble empirical mode decomposition (MEEMD) and adaptivefractional particle swarm optimization radial basis function neural network (AFOPSO-RBFNN)is presented. Firstly, the MEEMD method is employed to decompose the traffic data sequenceinto intrinsic mode function (IMF) component. Then, the AFOPSO-RBFNN is adopted toforecast the IMF components. Ultimately, the final prediction value is obtained via synthetizingthe prediction results of all components. The forecast results on real network traffic show thatthe proposed algorithm has a lower computational complexity and higher prediction accuracythan that of EMD and Auto Regressive Moving Average (ARMA), EMD and Support VectorMachines (SVM), EEMD and Artificial Neural Networks (ANN) method.4. Aiming at the high-dimensional nonlinear behavior of network traffic on small-timescale, a novel quantum neural network (QNN) model is presented. The quantum neural networkis composed of quantum bits, universal quantum gates and quantum weighted. Then, toaccelerate the convergence speed and prevent the algorithm from falling into local optimum, alearning algorithm based on modified descent Polak–Ribière–Polyak conjugate gradient(MPRPCG) method is given, and its global convergence is proved in theory. Forecasting resultson real small-time scale network traffic demonstrate that the proposed method has lowercomputational complexity and more accurate prediction precision than that of flexible neuraltree (FNT) and local support vector machine regression model (LSVMR). Moreover, comparedto BP neural network and Quantum weighted neural Network (QWNN), the convergence andthe robustness of the method in this paper are outstanding.5. Aiming at the difficult problem of determining the feature subset used to detect anomalyin machine learning task, an anomaly detection model based on normalized mutual informationfeature selection (NMIFS) and quantum wavelet neural network (QWNN) is presented. Firstly,in order to realize the effective reduction for high-dimensional feature data, NMIFS method isused to select the best feature combination from a given set of sample features. Then, the bestcombination of feature vectors are sent to the QWNN classifier for learning and training in thetraining phase, and the anomaly detection model will be obtained. At the detection stage, thedata is fed into the detection model that has been established during the detection phase; ultimately output the accurate detection results to the client. Considering the empirical risk andconfidence risk comprehensively, the learning algorithm of structural risk minimization extremelearning machine (SRM-ELM) is employed by the QWNN classifier. The experimental resultson real abnormal data demonstrate that the NMIFS-QWNN method has higher detectionaccuracy and lower false negative rate than existing common anomaly detection methods.Further more, the complexity of the algorithm is low and the detection accuracy can reach up to95.8%.Finally, the anomaly detection scheme, which is consisted of coarse detection technologybased on traffic prediction method and fine detection technology based on machine learningmethod, is proposed. Experimental results on synthetic data and real backbone traffic data showthat the detection accuracy of the proposed program can reach more than96.9%.