Research On Network-based Detection Algorithms For 0-day Polymorphic Worms

0-day polymorphic worm is one of the most serious threats to the Internet. Sinceit attacks unpatched software vulnerabilities, and presents di?erent byte sequence dur-ing each infection, it can damage a large number of hosts while evading detection.This dissertation focuses on the problem of network-based detection of 0-day poly-morphic worms. According to the potential techniques applied by polymorphic worms,researches in this dissertation are carried out based on three di?erent basic ideas, in-cluding port scan detection, worm signature generation and shellcode behavior detec-tion. For each idea, corresponding model and detection algorithm are proposed andevaluated through experiments. The main work and contributions of the dissertationare as follows.Port scan detection. It is di?cult for previous port scan detection methodsto distinguish new worms from great number of scan records. In this dissertation,a clustering-based method for port scan analysis is proposed, including scan vectormodel, in which any scanner's behavior is represented by a vector changing constantly,and a center-based clustering algorithm which works on those changing vectors. Theresult of experiments shows that the proposed method can identify di?erent worm scan-ning activities e?ectively.Worm signature generation. Signatures generated automatically from a fewworm samples can be used to detect worms. However, current content-based signa-tures can be easily evaded by polymorphic worms. Motivated by the characteristics ofbu?er over?ow attack, the length of protocol field is proposed to be the signature inthis dissertation. Based on the field hierarchy model of ?ows, length-based signatureis formally defined. Then the dissertation proposes a length-based signature generationalgorithm, and proves its false positive and false negative upper bounds, even underattacks. The experiments validate the performance of the proposed algorithm on everyaspect, especially the high accuracy under the worst-case attack. Shellcode behavior detection. Polymorphic shellcode is an essential ingredientof a polymorphic worm. In order to improve the speed and attack resilience of previ-ous detection algorithms for polymorphic shellcode, this dissertation proposes a noveldetection algorithm. As the basis of the algorithm, firstly the problem of polymorphicshellcode behavior detection is formally defined and analyzed, and then behavior de-scription model is built, in which behavior pattern is defined for the description andidentification of executable behavior. Furthermore,for the proposed algorithm whichis based on CPU emulation technique, a series of original techniques are designed andapplied. The experiments show that the proposed algorithm highly outperforms theprevious methods in both speed and attack resilience.As parts of the work on the three problems above, prototype systems have beenimplemented correspondingly. The experiments are all performed with the prototypesystems, using the dataset gathered from real network environment, including the tra?ctraces and the exploit code.