Data Minging Technology Applying To Intrusion Detction System

With the popularity of Internet, network security problem has become increasingly serious. In order to establish a comprehensive security defense system to solve this problem, various security devices, such as firewall, IDS, anti-virus tools and VPN, have been introduced. However huge amounts of raw attack alerts will be generated when these devices run. As a result, it is difficult for network administrators to find real security risk as well as the root cause of these risks in the network and respond correspondingly. Among these security devices, the intrusion detection system has been widely used in modern enterprise networks due to the fact that it can discover hacker's intrusion pattern in real-time. And it is very helpful to find the root cause more efficiently and discover novel and meaningful alerts by means of applying data mining to the field of intrusion detection system.In this paper, we focus on how to apply the data mining technology such as cluster analysis and association analysis to the intrusion detection system. In the field of cluster analysis, this paper make two improved algorithm named Partial-Sort-CLARAty and Roll-Back-CLARAty based on the original CLARAty algorithm. While in the field of association analysis, this paper put emphasis on the explanation of the Apriori and FP-Growth algorithm. On the basis of the two algorithms, we put forth two kinds of improved algorithm named PS-Apriori and Twice-Sort-FP-Growth and design the cluster-mining/association mining accordingly. The alert clustering method groups raw alerts by the form of generalized alert, and helps to identify the sharing root cause of these alerts. The alert association method exploits frequent security alert rule and discovers interesting strong rules as well as analyze the cause of the attack. Our two schemes exhibit preferable application effects when implementing with the MIT Lincoln Laboratory Scenario DDOS1.0 datasets. After making profound explanation on the data mining technology and intrusion detection system, the paper mainly focus on the following five parts to achieve innovation:1. Standardizing the raw alert dataset. Data mining technology requires the data source should have a united form before it is used for analysis. So before the further research, it's necessary to standardize the raw alert data source.2. Applying the clustering mining technology to the intrusion detection system. Put forth two new kinds of cluster algorithm named Partial-Sort-CLARAty and Roll-Back-CLARAty based on the original CLARAty algorithm. Specific explanation has been made on the over-generalization phenomenon during clustering. Deep comparison is also made among these three algorithms.3. In order to strengthen the productivity and normativity of the generalized-rules, a unified XML-pattern document has been designed to encode the high-level alert property. It is also used to describe the conceptual-hierarchy of the corresponding alert property. In the lab test part, it is also used as a standard for the cluster mining.4. Applying the association mining technology to the intrusion detection system. After explaining the weak point of the original Apriori algorithm and FP-Growth algorithm when applying to the intrusion detection system, two improved algorithms have been put forth, which, respective named PS-Apriori and Twice-Sort-FP-Growth algorithm. Comparison has been made between these algorithms and implementation has also been made.5. Conducting test based on the popular test dataset and make analysis on the test result. Comparison has been made between the test environment and the real topology. Deduce the more general result from the result set and the test is successful.