Study Of False Positive Reducing And Alert Clustering In Large-scale Network

Intrusion detection is a paramount technique for network security defense. But the security data produced by relevant products is low-level, besides it will take large amount of human cost. In large-scale network the quantity of security data is huge, more than 100,000 alerts will be produced per hour by Intrusion Detection System in 100Mbps access network. It is impossible for administrator to handle it in good time. Though some security events may result in a few alerts, they will be inundated with large numbers of redundant alerts. How to reduce them in real-time has become a problem required to be solved urgently for large-scale network security protection.Alert is composed of multiple attributes, such as signature, source IP, destination IP etc. False positive refers to these alerts which are misjudged by security rules when match with packets produced by normal activities. They are quite similar with true alerts, what's more, the quantity is huge. it will make further alert correlation and attack discovery confront enormous difficulties. False positives can be reduced by three measures: rule optimization, periodicity analysis and alert clustering, which can diminish the quantity and improve the quality.Intrusion detection rule not only can produce true alerts but also will trigger false positives. Security cost of some rules will be increased dramatically because of large numbers of produced false positives. Tradeoff between security cost and yield will be evaluated from the economic utility in the model for rule optimization. Rules which have greater FPP (False Positive Probability) will be deleted according to the security requirement of protected object. In the meantime, rule FPP network can be set up based on Bayes analysis. Then alerts produced by the rule will be classified by prior information and FPP can be diminished.Through investigating alerts produced by Network Intrusion Detection System in large-scale network, some specialties can be discovered: the distribution of some attributes has obvious heavy-tailed characteristic, most alerts come forth with a remarkable periodicity, what's more, most attributes of periodic alerts concentrate on the head of heavy-tailed distribution, and these attributes appear uninterruptedly. The algorithm for recognizing alert stream can find the combination of attributes which produce a chunk of alerts based on heavy-tailed distribution. The algorithm for reducing false positives can discover the period of every alert stream by auto-correlation analysis and Fourier analysis, and which will be validated by F-test. In experiments more than 62.5% of raw alerts can be filtered, and all these filtered alerts are false positives by some deeply relevant investigation.According to the source-to-destination pattern of produced alerts, security events can be classified into 3 types: multiple to one, one to multiple and one to one. According to these 3 principles the threshold of time and number can be set, afterwards multiple alerts can be clustered into one meta-alert dynamically. Meanwhile the type of security event will be distinguished, two indexes will be proposed to evaluate the reliability and emergency of meta-event. Experiments demonstrate that after clustering more than 98% of raw alerts can be diminished. In addition, anomaly detection can be implemented by Chebyshev inequality in raw alert series, filtered alert series, frequency series of alerts. Then the capability of detecting specific security events can be improved. The above researches will provide new theoretical foundation and methods for false positives reducing and security event correlation in large-scale network.