Real-time Network Traffic Detection And Analysis In A Distributed Environment

With the development of Internet and big data technology,while bringing great convenience to people in information utilization and resource sharing,various attacks and network anomalies emerge in an endless stream,which leads to a series of security problems.With the increasing diversification of network traffic,the requirement of real-time,reliability and security of network traffic for key traffic is becoming higher and higher.At the same time,more and more malicious traffic is gradually flooded in the network,which consumes a large amount of network bandwidth resources and seriously affects the transmission of normal network traffic.How to better monitor network behavior and achieve effective network resource management is one of the main problems faced by network practitioners.The efficient statistical analysis of network traffic and the ability to detect malicious or suspicious traffic can help administrators control network traffic more comprehensively,judge and respond quickly according to possible abnormal conditions,and thus improve the utilization rate of network resources.This topic mainly focuses on the research of massive real-time network traffic anomaly detection in distributed network environment,in order to deal with traffic data in real time,some related algorithms,such as data mining and machine learning,which are widely used at present,are not fully competent in real-time computing environment because of their high computational complexity or need to scan data sets many times.In real-time environment,we need to adopt simpler and faster models and algorithms,to solve this problem,we need to establish a dynamic network traffic detection and analysis model to perceive the changes and trends of network traffic through the study of network traffic analysis methods and the analysis of various traffic flows in the network.This paper focuses on the design of traffic characteristic attribute measurement method and detection model.On this basis,a complete detection system is preliminarily realized,it tries to identify unknown and suspicious network behaviors through fast anomaly detection algorithm,in addition it realizes the detection of known threats through pattern matching.In this paper,three new methods for measuring traffic attributes are proposed,which are based on triangular tangent correlation and based on cumulative variance and based on divergence-weight.The three methods are analyzed in detail with the measurement of information entropy.Based on the design of measurement method,two detection models are designed.One is a simple and fast threshold updating model based on exponential weighted moving average algorithm,the other is a more general dynamic adaptive updating model based on deviation degree.Experiments and applications of the two models are discussed respectively.At the same time,combining the advantages of the existing traffic model,the detection model and scheme for real-time network traffic are realized,and the related contents of network traffic collection and log processing are implemented in detail.On this basis,a real-time,efficient and fast detection and analysis system which can run in the distributed network environment is constructed with the combination of distributed architecture and large data processing platform.The system and deployment plan can realize real-time perception of network traffic changes,anomalies and trends.