Research On Network-based Automatic Attack Signature Generation

A signature-based detection is the most common and effective way to detect attacks due to its simplicity and online response. The efficient and accurate signature generation is critical in the signature-based detection systems. So far, the signatures that used by signature-based intrusion detection systems (IDSs) are produced manually by security experts, a process too slow. In this way, signatures can be only provided by security experts after a worm has already attacked systems and caused damages, which will miss the best defending time. So, they do not fit for the requirements for the Internet safety, since the new attacks nowdays are produced timely and the spread speed of worms is extraordinarily faster than human beings can respond. Besides, the polymorphism techniques can be used by attackers to evade detections. To support the automatic and speedy generation of signatures, a number of automatic signature generation approaches and systems has been proposed. These approaches and systems can be broadly classified as either the host-based or network-based. The Network-based signature generation (NSG) systems produce the content-based signatures only through analyzing the suspicious network traffics. And the host-based signature generation (HSG) systems generate the signatures based on the informations get from the protected hosts. Our researches systematically study NSG techniques and their applications. Especially we deeply study the signature generation for polymorphic attacks, such as polymorphic worms. The main contributions of this paper are summarized as follows.(1) A new signature type—SRE (Simplified Regular Expression) signature is proposed and the NSG problem is modeled. SRE signatures can be easily transformed to the rules in current IDSs to accurately detect attacks. Based on SRE, we provide formal definitions of what is "a more specific signature" and what is "the most specific signature" of a polymorphic attack such that we can compare the accuracy of two SRE signatures. We prove that the most specific signature generation of a polymorphic attack is NP-hard.(2) Noise filtering methods for attack sample collection are proposed. To capture the samples of new attacks for signature generation, we design and implement a distributed Honeypot system—HonIDS. In contrast to the traditional approaches, which take all traffics visiting the honeypot as attack samples, we propose to filter the noisy attack samples out of the traffics visiting the honeypot, where a noisy attack sample means a network flow from a benign user instead of an attacker. Two detection models are proposed and integrated in HonIDS, TFRPP model and Bayes model. Based on these two detection model, we propose three methods for filtering noises from attack samples.(3) Signature generation methods based on multiple sequence alignment are proposed. The generated signatures by previous NSG systems are not accurate enough since two kinds of information are lost. First, some invariant parts in polymorphic worms can not be extracted, like one-byte invariant parts. Second, all distance restrictions between invariant parts are neglected. Referring to some related algorithms in bioinformatics, we propose a signature generation approach based on multiple sequence alignment (MSA). Motivated by different signature generation applications, we propose a series of sequence alignment algorithms, including the CSR algorithm and the ECSR algorithm for pairwise sequence alignment, the MSA algorithm HP_MSA for noise-sensitive signature generation, and two MSA algorithms HP_MSA and T-Coffee+CSR for noise-tolerant signature generation. Experiment results show that our signature generation approaches based on multiple sequence alignment can produce more accurate and precise signatures for polymorphic attacks, comparing to previous approaches.(4) The idea of signature tree and an incremental signature tree generation approach are proposed. We observe that signatures from worms and their variants are relevant and a tree structure can properly reflect their polymorphism relationship. Rather than generating isolated signatures for multiple polymorphic worms in current NSG approaches, we propose to use the "more specific than" relation to organize generated signatures hierarchically into a tree, so-called signature tree. In this signature tree, each node is labeled with a signature and a signature of a child node must be "more specific than" the one of its parent node. The signature tree gives insight on how the worm variants evolve in time, and makes it simpler to balance the false positive rate and generalization ability of signatures and makes it easier to organize and maintain the generated signatures. The most complicated situation of signature generation is when the suspicious flows captured by an NSG system contain mixed samples of multiple polymorphic attacks (perhaps accompanied by noises). Based on the idea of signature tree, we propose an NSG system—PolyTree, which uses the ISTG algorithm to incrementally generate a signature tree for multiple attacks. Upon encountering a new suspicious flow, the ISTG algorithm will be called to generate more specific signatures using the PDRP_MSA algorithm in a fixed signature tree and to update this signature tree. Experiment results show that the generated signature tree through the ISTG algorithm has two significant properties. First, the samples from the same attack can be well clustered into one node in the signature tree; second, the final generated signature tree contains the most specific signature for each encountered polymorphic attack given adequate worm samples collected from it. This thesis proves the correctness of ISTG algorithm and analyzes potential malicious attacks on ISTG algorithm.(5) In order to integrate the algorithms and techniques presented in this thesis, an NSG application system is designed. In this design, we focus on security collaboration. Since there is no unified model to ensure interoperability and collaboration within different security components and systems, we first propose an abstract-level security collaboration model BSCM (Blackboard based Security Collaboration Model). In this model, network security components don't directly communicate with each other, but via a common blackboard which serves as the platform of information-sharing and events-responding. Based on BSCM model, a distributed NSG application system is designed.