Large-Scale Network Abnormal Detection Based On Graph Model

With the increase of the network hosts and applications,it is becoming more and more challenging to identify abnormal traffic in large-scale networks.At present,there are many useful methods of analyzing abnormal traffic both domestic and overseas,focusing on the analysis of specific nodes in the network.Such methods cannot cope with the need of network management and security monitoring in large-scale network.Therefore,an effective network group partition method is urgently needed to cope with the challenges of large-scale network.In this paper,a method is proposed for large-scale network abnormal detection based on graph model,which provides a new way to solve the problem of large-scale network abnormal detection.First,the bipartite graph model is constructed on the basis of the IP field in the NetFlow data.Second,the single mode projection graph model is developed based on the relationship between the number of common nodes and the similarity of the host.Third,The Louvain community detection algorithm is implemented by Spark GraphX to detect the constructed graph model in the community to find the host group with similarity.Finally,the traffic pattern within the community is analyzed based on the relative uncertainty model,the TCP flag,and the depth data packet.Meanwhile,a system is implemented for large-scale network abnormal detection based on graph model.It is designed from three aspects which contain data acquisition,storage and computation,ensuring the scalability of the system.Firstly,the Flume distributed log collection component is used to collect and transmit the data in real time.Secondly,HDFS and HBase are used to store historical data and the results of calculation.Finally,the Spark cluster is used to complete the analysis of the historical data of the massive NetFlow.Finally,the method and system of large-scale network abnormal detection based on graph model provide a new direction for large-scale network abnormal detection.The experiments show that the validity and extensibility of the proposed method are verified by the experiment and analysis of the NetFlow data of Tianjin University of Technology.