Analysis And Detection Of Botnet Anomaly Traffic

The botnet is used by the hackers to launch the DDoS attack, send spam, steal the financial information and so on, which does great harms to national network security. So how to achieve the anomaly detection of botnets urgently remains to be solved.This thesis obtains the suspicious IP through analyzing the anomaly traffic collected by honeypot, and analyzes on the daily number of suspicious IP and the statistical characteristics of anomaly traffic in detail, so that the traffic of the high speed network is filtered using the obtained set of suspicious IP to reduce the traffic effectively. Meanwhile, this thesis improves the efficiency of the signature extraction through research on the signature extraction technology, to enhance the performance of botnets detection algorithm based on signatures and accomplish the detection of known botnets better. In addition, this thesis combines the botnets detection algorithms respectively based on signatures, flow characteristics and the behaviors, in order to detect the unknown botnets more accurately. The main contents and innovations of the thesis include the following topics:Anomaly traffic analysis and the experiment statistic model based on honeypot. This thesis carried on a 470 days'uninterrupted monitoring on the honeypot implemented in the data center and obtained the distribution statistic model of the everyday number of anomaly IP. Besides, it pointed out that the number of scan IP per day follows the Normal distribution in a short period. The distribution statistic model of the number of anomaly IP can provide an experiment statistic foundation for the assessment of the scale of the suspicious IP of the botnet. Meanwhile, this thesis makes a detailed analysis of the attack frequency of the attacker, the attack frequency of the port, and so on. The analysis pointed out that the network administrators should filter the suspicious IP, ports, and suspicious network, and then make the limited security resources focus on the suspicious targets to solve the network security problems better.Research in Automatically extracting Signatures of the Botnet. The signature extraction algorithm can implement the auto-extraction of the signature, providing the foundation for the botnet detection. But the botnet has its inherent traffic characteristic and the present algorithms of extracting characteristics of the signature can't suit the botnet detection completely. In this thesis we proposed an automatical extracting signatures algorithms and system designing framework based on the known algorithms and made improvements according to the distribution of the botnet signature characteristics, which can self-adaptively extract the signature characteristics of data flows of different functions. The improved botnet signature extraction algorithm is used to extract signature of the data collected from the botnet anomaly traffic, the results of the experiment suggest that, the improved algorithm acquires far more effective signatures than before therefore it can finish the anomaly detection of the botnet better.The research of botnet detection algorithm based on the behavior characteristics are mainly as follows:1) The research of the botnet propagation characteristics. The process of worm propagation can be analyzed based on epidemic model according to the different network. The propagation process of the botnet is different from that of worm, and the infected hosts of botnet accomplish the cooperation according to the different tasks. The hosts used for botnet propagation are only a part of the infected hosts and the propagation process of botnet is controlled. Based on SIR model, this thesis models according to the characteristics of botnet propagation and analyzes the relationship between the number of infected hosts and spread proportion. Meanwhile, by introducing the definition of the spread duration, the network stability can be analyzed quantitatively. The model theoretically analyzes the relationship among the spread proportion, the expected botnet scale and the network stability, and points out that the characteristics of controlled botnets can better adapt to the development trend of the increasing decentralization of the botnet scale.2) Research for scan detection algorithm of high-speed links. In order to detect the scan attack on high-speed links effectively, this thesis improves the scan detection algorithm TRW based on honeypot, and makes a detailed analysis on its performance. The analysis shows that the improved algorithm has better performance on the speed of identifying the scan source. Meanwhile, on the basis of selective system sample, this thesis focuses on the analysis of the anomaly detection accuracy of three scan detection algorithms:Snort,TRW,TRWHP. The experimental results show that, at the same sampling ratio, the false positive rates of TRWHP and TRW algorithm are almost the same, however, the false negative rate of TRWHP algorithm can make a remarkable improvement and obtain better detection performance. Meanwhile, to the known selective sampling method based on the length of the packet, this thesis analyzes its influence on the performance of the scan detection in theory. At the same time, this thesis compares the detection performance of the scan detection algorithm TRW in the selective sampling method based on the packet length and the system sampling method. The experiment results show that on the condition that the packets number is equal before and after sampling, the selective sampling method based on the packet length generates less influence on the detection performance of TRW and can obtain high accuracy of scan detection. The research which is about the influences of various sampling methods on the scan detection algorithms provides a basis for the anomaly scan detection in the high speed links.3) Anomaly Detection of Botnet based on Multi-level Classifier. The botnet variants emerge in endlessly. Although the known anomaly detection algorithms can detect the unknown botnet to some degree, the generalization and the accuracy of the models remain to be improved. A botnet anomaly detection structure based on multi-level classifier is proposed in this thesis. The first level classifier proposes a new periodical communication detection method. Compared with the spectrum analysis method, this algorithm has lower complexity and can make both online and real-time detection of the botnet. The second level classifier uses the decision tree algorithm to detect the botnet, based on the statistical characteristics of the IP pairs of the periodical communication. The result of experiment indicates that the periodical communication detection algorithm can obtain better accuracy of anomaly detection, but its false positive rate is higher. Compared with the algorithms that only adopting the periodical communication, the multi-level classifier model can obtain the smaller false positive rate(6.5%) on the condition that the accuracy of anomaly detection is the same. Therefore, the multi-level classifier model has higher performance in detecting the unknown botnet.The research on the botnet detection based on the flow statistic characteristics. This thesis combined the detection method based on the fingerprint characteristics with the detection method based on the flow characteristics and proposed two-level botnet detection structure to achieve the IRC botnet anomaly detection of the high speed network traffic. Firstly, it implements the accurate detection of the IRC network traffic based on the signature of IRC protocol. Then, it takes advantage of the characteristics selection algorithm to accomplish the characteristics subset. Besides, in view of different characteristics subsets, it makes use of LADTree classifier algorithm to implement the botnet anomaly detection. The results of the experiment indicates that taking advantage of LADTree classifier on the basis of the different characteristics subset(subset 1 and subset 2) and detecting the botnet in view of data sets of different periods, the anomaly and normal detection accuracy are respectively over 83.3% and 93%. This indicates that the proposed two-level botnet detection structure has good detection results and generalization ability.