Research On Anomaly Detection Based On Data Mining And Multi-stage Intrusion Alert Correlation

The emergence of the network provides a convenient platform for communication. When enjoythe service, we have to caution how to protect the information's integrity,privacy and availabilityeffective. This refers to the security protection of information system. The security protectionsystem is a kind of defense system, which includes four levels: protect, detect, react and recovery.Intrusion detection system is one of the most important components of safe system architecture,playing the role of "Early-warning system" in computer application environment. By collecting theinformation of operating system, system program, application program and network packages,intrusion detection system discovers behaviors that against security policy or endanger the system.The system with intrusion detection function is called Intrusion Detection System, or IDS for short.IDS is different form other ordinary information system, its security policy require the integrity ofdata. The process of deal with mass of alert information is even more difficult than analysis thealert correlation, what's more, the increasing calculation presents a higher demand for hardwaresupport, which greatly increased the application cost.From the report of Symantec and other security agencies, the network security situation isdeveloping form "pure" trojan virus which attack independent or irrupt single computer to attackcollective based on "Internet + attacking team + virus + commercial profit" .With new kind ofattack and complex multi-step attack in endlessly in recent years, most of existing IDSs do not havemulti-step attack detection characteristics. Therefore, the identification of unknown attacks andmulti-step attacks have been researched deeply by expert at home and abroad, and have got someachievements in alarm correlation(alarm correlation is smaller than alarm correlation). Concerningabout the current situation of alarm correlation and multi-step attack research and application, wecan elaborate in two aspect, online and offline, as follows:(1)Online alarm correlation system, in order to improve data compatibility, integrate the alertresource. Thus lead to low detection rate and high false alarm rate. Moreover, there are seldomresearch outcomes about multi-step attack.Offline alert correlation detection rate is higher, but consumption of system resource isconsiderable, using in real application is not suitable.The main research of this paper is how to improve the detection rate and reduce false alarm rateabout a single alert data and identification of multi-step intrusion after correlation analysis of multiple alerts. The main research of this paper is four aspects as follows:1. In order to improve practical research, we propose anomaly detection based on data mining andmulti-step intrusion alert correlation model, Comparison with domestic and foreign researchresults, in this section, we integrate the relevant methods of data mining and propose theapplication model that can be used and recommendation.2. In order to improve the efficiency and effectiveness of alert data analysis,we focus oncompression technology about the attributes of the data, According to the classical method ofthe key attributes about the data mining analysis, Using standard data set of network intrusionand detection characteristic in the intrusion detection environment. We propose characteristicanalysis method about key attributes for intrusion detection and give experimental results.3. In order to improve the detection results of intrusion detection, we focus on research ofanomaly detection and noise data analysis method. According to the characteristics of intrusiondetection data and the need for real-time detection, we propose optimization strategies andmethods of pruning for real-time intrusion detection and give adequate experimental. And withexisting research for the field of anomaly detection have been compared and discussed. Thepractical value and shortcomings of this method is verified by experiment. Provide futureresearchers with research ideas about this field and experimental data for reference.4. In order to achieve collaborative attack (or multi-step attack), we focus on the data filtering,data fusion and association analysis of the data, using filtering preprocessing of single intrusionalert, integration of multi-source intrusion alert and correlation analysis of multi-step intrusionalerts. According to the characteristics of intrusion detection data, we propose filter based onthe probability of false positives, the alert fusion based on fuzzy mathematics, probability andfuzzy-based multi-step method of intrusion alerts correlation and based on logic descriptionlanguage description of the multi-step intrusion alerts. At the same time relevant to this field ofexiting research have been compared and discussed. This method is verified usingexperimental data by experiments. Provide a reference for the future multi-step intelligentintrusion detection research and related data, Provide practical ideas, experimental methodsand results for the future research study.Four major innovations:1) This paper presents anomaly detection and a framework for multi-step alerts correlation modelbased on data mining. The framework proposed is under the application point of view, which hasgood scalability. Our work is carried out base on it.2) We proposed attribute of key characteristic which is Applicable to the field of anomalydetection. Compared with foreign Characteristic selection results, our attribute is subset of theirs.The experiment on KDD1999 datasets shows that our technology can reduce 7%-16% detectiontime meanwhile only a mall impact on accurate. so this will improve the performance of real-timeanomaly detection. 3) We also Proposed a real-time anomaly detection algorithm ADTSFC(anomaly detection withtime-stamp frequent pattern-based clustering)which is based on BIRCH and ADWICE. Resultsunder KDD1999 datasets reflect that false positives can effectively reduced while detection rate cangreatly improved using this algorithm.4) The description of PDDL (Planning Domain Definition Language) of the field of intelligentplanning on multi-step intrusion detection. And it have been facilitated the extended of intelligentplanning on intrusion detection area. (The similar research had not been found before our paperfinished).Our work is carried out under the concern about the demand of the development of networksecurity both at home and abroad. Our team has always been concerned and carried out datamining-based intrusion detection, and gain some achievements. We hope to cooperate andcommunicate with other researchers, their concerns mean a lot to us.