Network Attack Scene Reconstruction System Based On Large Data Analysis

With the frequent occurrence of network attack events and the diverse means of invasion,Intrusion Detection System(IDS)generates massive redundant and false alert data every day.By analyzing the isolated alarm information,we can restore the attack process and help the administrator find the weakness of the network.In this paper,we use the key technologies of Spark and Hadoop to implement the complete architecture of data collection,storage,data transmission,calculation and display,which can effectively deal with the problem of massive alarm data.Based on the architecture,we analyze the network alarm log by alarm association methods.We designed and implemented a system to rebuild the intrusion scenarios.By studying and comparing the existing methods of alert correlation,we focus on the research of the two classic methods,the first one is “Causal Correlation” and the other is “Probabilistic Correlation”,and we also implement both methods on our system.For the first method,we introduce the alert fusion preprocessing module to reduce the redundancy of original alerts,and the resulting attack scene graph becomes more clearly.For the second method,Probabilistic Alert Correlation,we defined the similarity functions of IP address,port and alarm class.We also give the definite minimum similarity expectation and weight of each attribute according to the experiment so as to calculate the overall similarity of two alerts.The whole attack scene reconstruction model is divided into three separate modules to be processed,including the redundant module to remove duplicate alarms,the aggregation module to aggregate each attack step,and the association module to form a multi-step attack scene.In order to verify the validity and practicability of the system,we experiment base on the real alert data set and the DARPA data set.The results turned out that our method can show the network attack step more intuitively than the result of “Causal Correlation”.This system can reduce the redundant IDS alerts,accurately associate similar alarms,as well as enhance the availability of IDS system.