Network Situation Analysis Framework Based On Big Data

With the rapid development of computer and communication technology,network attack behaviors are developing towards the trend of distribution,scale,and complexity.Relying on a single network security protection technology such as firewall,intrusion detection,antivirus,and access control,can no longer meet the network need for security.Therefore,new technologies are urgently needed to detect abnormal events in the network in a timely manner,to sense the network security status in real time,and to make predictions and early warnings to reduce network security risks and improve network protection capabilities.In this thesis,multi-source heterogeneous massive situational data,especially network traffic multi-dimensional characteristic data,is the main body.The key technologies of multi-source heterogeneous data fusion and correlation analysis,situation assessment and prediction methods are studied,and a new method based on big data Network situation analysis framework,designed and implemented a prototype system.The main work and contributions of this thesis are as follows:(1)Researched related technologies such as situation definition and quantitative expression method,situation data correlation analysis and situation assessment method,situation prediction method based on big data,and proposed a network situation analysis framework based on big data.The framework combines technologies such as analytic hierarchy process,multi-dimensional index abnormal aggregation technology,ARIMA,etc.,to realize the overall planning of situation data collection,analysis,situation assessment and prediction in the big data environment,and finally realize the corresponding network security situation awareness system support platform.(2)A situation data acquisition framework based on flume + kafka + spark streaming is designed.Flume+kafka is used to extract the logs generated by the external third-party situation data generation system(such as traffic analysis equipment),which flows into spark streaming in the way of data flow,and realizes the real-time situation data acquisition.(3)Based on the classic evaluation framework model given by the predecessors,this thesis summarizes its shortcomings,combines its advantages,and puts forward two kinds of evaluation models: the situation value is obtained through the evaluation method based on AHP,at the same time,the abnormal index is correlated through the situation evaluation model based on FCE-AHP,and the abnormal grade is obtained.(4)This thesis has carried out in-depth research on various types of prediction algorithms.Aiming at the shortcomings and deficiencies of traditional network traffic prediction technology,a combination of signal decomposition technology in the field of digital signal processing,as well as signal period analysis,component regression prediction and other technologies are designed.A new prediction algorithm based on network traffic characteristics was proposed.(5)Through the research and test of network situation framework,this thesis designs and implements a prototype system based on spark.The prototype system consists of four functional modules: situation data acquisition and storage analysis framework;situation assessment module,situation prediction module,situation data cache and retrieval module.(6)The main task of this thesis is to realize the master plan of situation data collection,analysis,situation assessment and prediction under the big data environment,and to realize the integration and collaborative mechanism design of various key technologies.Adopt spark memory database,read the data stream in real time,analyze the situation in real time,improve the real-time performance of the system.After the experiment and system test,the usability and accuracy of the system are verified.