Research On Key Technology Of Network Traffic Data Processing And Anomaly Detection

With the information network continues to be deepened in social production and life,the harm caused by network attacks has been expanded dramatically.As an effective way to discover network attacks,network traffic anomaly detection can identify suspicious flow from the traffic,which has become a hot research topic in the field of cyberspace security.Network traffic data processing can reduce the amount of data awaited detection and condense the information contained in the data,thus improve the detection efficiency and accuracy of network traffic anomaly detection.Therefore,this thesis studies the key technologies of network traffic data processing and anomaly detection,the main works are as follows:1.For the traffic collection demand of network traffic data processing and anomaly detection,the traffic collection method is designed and implemented based on the NDIS(Network Driver Interface Specification)intermediate driver.First,we compare and analyze the advantages and disadvantages of some typical traffic collection methods of Windows system.According to the high efficiency and unavoidability of the NDIS intermediate driver,it is selected as the traffic collector.Then,the traditional NDIS intermediate driver and the modern NDIS filter driver are implemented.For the different processes of sending and receiving packets,the corresponding processing service of the driver are selected to collect and analyze the packets.Finally,the process of submitting traffic data is designed.To solve the problem that the traffic submission and collection processes constitute a loop,the flow is differentiated according to the characteristics of the submitting process.A public server and message middleware are designed and used for communication coordination in the submitting process,which could facilitate the subsequent traffic processing and detection.2.In order to increase the classification speed of the ABV(Aggregated Bit Vector)algorithm,this thesis proposes a connection-oriented algorithm of IABV(Improved Aggregated Bit Vector).Based on the characteristic that the packets with same connection have similar classification results,the IABV algorithm establishes a Hash table-rule set two-level searching structure.It first searches in the Hash table to check the packet classification rule.When the Hash table lookup fails,it go to search the matching rule in the rule set.To avoid the accumulation of rules in the table,we propose a collision handling mechanism by utilizing the time-limit features of connections.When a Hash conflict occusrs,it judges whether to overwrite the Hash table entry or not according to the last hit time of the entry.Secondly,to accelerate the rule set searching,the IABV algorithm divides each dimension into multiple intervals equally and employs an array to index these intervals.Finally,in order to reduce the time and memory consumption of the algorithm,the prefixes of the rule set are converted into range to reduce the complexity of the search structure.The experimental results show that the performance of the IABV algorithm is improved by converting prefix into range and the time performance of the IABV algorithm is significantly improved compared with the ABV algorithm under the same conditions.3.To quickly and accurately select high-quality network traffic anomaly detection feature set,a hybrid feature selection method is proposed.Firstly,to reduce the amount of calculation and identify the redundant features,we define the ratio of mutual information between a feature and other features to the feature entropy as the redundancy degree of the feature relative to other features.If the ratio is greater than a predefined threshold,the feature is judged as redundant and should be deleted from the feature set.Secondly,we propose an assessment mechanism for valuing the features to accurately measure the impact of each feature on network traffic anomaly detection.This method uses the ratio of the anomaly detection accuracy after and before delete a feature from the feature set to measure the effect of the feature on detection.Then,the features are sorted according to their importance and the top k features with the highest detection accuracy are selected as the result.The experimental results show that the proposed method can quickly screen out a feature subset with good detection performance and lower dimensions.4.In order to increase the detection accuracy of fine-grained network traffic anomaly detection,a network traffic anomaly detection method combining serial and parallel processing based on feature selection is proposed.To improve the detection accuracy,the method proposes a two-level structure.It consists of abnormal traffic coarse-grained detection and abnormal traffic type fine-grained detection.The structure could use the best feature set of each type of traffic for anomaly detection according to their characteristics.Firstly,it uses a binary classifier for coarsegrained detection to divide the traffic into normal and anomaly quickly.Then,we combine multiple binary classifier parallel detection and single multi-classifier detection to identify the type of abnormal traffic.It can avoid the problem of error accumulation in serial detection method,improve the detection accuracy,and reduce the detection time.Finally,NSL-KDD data set is used to verify the proposed method.The results show that the proposed method can effectively improve the detection precision rate and recall rate of each type of traffic.5.To solve the problem that massive traffic data is difficult to process and detect in real time,we design and implement a network traffic data processing and anomaly detection prototype system by adopting the Storm distributed processing platform.The system can be divided into three modules: network traffic collection,network traffic feature extraction and network traffic anomaly detection.Firstly,it uses the NDIS intermediate driver to collect data and uses the packet classification algorithm to filter the packets quickly.Secondly,through the coordination of a public server,the traffic waiting to be detected will be submitted to the feature extraction topology that running on the Storm platform.The topology extract the traffic features to construct a feature vector.Then,using the detection model that has trained offline and the feature set which is selected by the feature selection algorithm,the anomaly detection module perform anomaly detection on the feature vector.Finally,the experimental results have verified the effectiveness of the prototype system.