Research Of The Real-time Network Data Flow Anomaly Detection Based On Storm

With the rapid development of Internet industry and the increasing of Chinese Internet users, the network security problem gradually getting people's attention. Network has penetrated into the life of people, and all kinds of network security events related to people's personal information and property safety. For all kinds of network security events, if they can be found in time, then we can take effective measures to reduce the damage even avoid loss. Otherwise, the amount of network traffic data is large now, how to calculate effectively in real-time is also need to be solved. In this paper, we research the feature selection technology, network data flow anomaly detection technology, and realize anomaly detection system based on Storm. Finally, the system is able to detect anomalies in the network.In this paper, for the problem of the complex situation of various network anomaly traffic characteristics,features of network data flow are selected. FCBF (Fast Correlation-Based Filter) feature selection algorithm is simply reduce the redundancy between feature and feature without considering the feature subset of the distinguish ability .The algorithm was improved by considering the feature subset evaluation criteria,which makes feature subset better.Aiming at the problem of abnormal traffic detection, this paper uses the anomaly model based on information entropy. Aiming at the problem that the Sketch data structure has low performance in the process of information entropy calculation and can not locate the abnormal point, we optimized the Sketch data structure. For the problem that the anomaly type is difficult to be determined in the anomaly detection model based on information entropy, the feature vector is defined and the anomaly type is determined by Euclidean distance calculation.Finally, on the basis of the above two parts, this paper implements the network data flow anomaly detection prototype system based on the Storm which is a distributed real-time computing system. Through the actual operation of the prototype system, it is proved that the system has a high processing performance and can detect abnormal data flow correctly.