Research And Implementation Of Advanced Persistent Threats Detection Method Based On Data Mining

In recent years,in the context of the rapid development of Internet communication technology,various enterprises and organizations have become more dependent on information systems.While various enterprises and organizations make use of information system,the vulnerability of information systems and the lack of security mechanisms have increased the risk of their important information assets being stolen.In this context,Advanced Persistent Threats(APT),the attacks targeting on confidential information of enterprises and organizations,including but not limited to stealing information and destroying important facilities,are increasing.APT attacks generally have the characteristics of hiding from the intrusion detection system,various attack means,clear target of attack,and always high attack cost(including time cost).Because of above characteristics,the traditional IDS is somewhat of limited effect in response to APT.Using all network traffic for analysis ensures that the network data of the attack is completely preserved.However,if all network traffic is used as the raw data for analysis,since an attack is likely to last long,detecting APT attacks from a huge mount of network traffic is also a big challenge.There are also many problems in the current research on APT detection.Since APT activities can hide from the detection of IDS by exploiting 0-day vulnerability,The APT detection method based on IDS alerts tends to lack of critical alerts corresponding to the vital steps of APT,which leads to a failure of detection.On the other hand,machine learning methods based on network traffic log tends to use supervised methods.However,supervised methods are more dependent on the quality of the training data and the same distribution of training data and prediction data.Considering that APT attacks often involve important information assets of organizations,while the amount of data involved is too large to label one by one,the acquisition of high-quality training data is not easy.In view of the above problems,this paper has conducted in-depth research on APT attack detection technology,and the research results are as follows:(1)Considering the amount of log is huge while the collection time window is large,we analyzed APT attack traffic log data and propose a traffic log reduction algorithm based on popularity and the direction of connection.Based on this,an efficient traffic log reduction technology is provided.(2)Considering the lack of labeled real data for APT detection,we did research on APT and proposed the detection model for the network behavior of C2 phase in APT.Firstly,for the C2 domain name access record,several features based on DNS behavior characteristics are proposed,and these features are merged with traffic features.Based on these features,the model uses the isolation forest anomaly detection algorithm to evaluate the data set.(3)An APT detection framework was designed and implemented,which contains the algorithm and the model above and has the ability of processing data online.In the framework,we proposed an online scheme of data reduction algorithm,which is able to expand and process flood traffic?The framework also provides an interactive interface allows the framework to flexibly configure parameters such as time windows and provides a friendly interface for security personnel.(4)The data collected by large-scale organization and the simulation data were used to verify the data reduction algorithm and APT attack detection model.The experimental results show that the algorithm has certain ability to detect APT attacks under large data volume,and can give Suspected infected host,providing a reliable starting point for further analysis and analysis.(5)Based on the above proposed methods,summarize the methods and frameworks,discuss the problems and propose the direction of further optimization.