Detection Model Based On The Simulated Execution Of Malicious Behavior

Internet worm has been one of the most significant threats that Internet faced, Compared with traditional viruses, which spread faster, more destructive. Polymorphic worm using Polymorphism techniques change form of the byte sequences to avoid signature-based Intrusion Detection System. How to quickly and effectively detect polymorphic worms is an important research direction of network security.This thesis based on the studies of the structure and the execution behavior of polymorphic worms, Reference Polygraph's Architecture, present emulation-based of the malicious behavior detection model. And introduce the principles and adopted algorithms of core modules in particular, The experiments verify the validity of the model and algorithms.The main work of this thesis includes following issues:(1) Malicious behavior detection based on emulation:According to analysis the execution principles of Polymorphic worms, this thesis adopts behavior-based detection methods, implement executable code of the network in a cpu emulator, and take GetPC Code and recycling decryption operations as acts of feature detect network data stream, differentiates suspicious data flow from normal data flow.(2) Clustering based on the maximum flow minimum cut theory:Network data flow detected by malicious behavior detection, As suspicious data flow may contains a variety of worm, even including the normal data. Therefore, it needs to remove noise and group the similar data by clustering. Then, deal with each group as a polymorphic worm and extract signature. In order to improve the efficiency of clustering, this thesis presents the theory based on the maximum flow minimum cut clustering algorithm. The algorithm uses undirected graph with weights to represent the relationship between suspicious data flow. On the basis, using Gomory-Hu algorithm to calculate the similarity of any two suspicious data and group the high similarity data. This algorithm is efficient and has the advantage of high clustering quality according to compared with hierarchical clustering algorithm of the Polygraph in theory.(3) Feature extraction:By malicious behavior detection and clustering,many types of worms filtered out. so as to suppress worm propagation timely, we need to extract signature for each class after clustering. this thesis extract all substring that satisfy limits of length and times from comparison sequence as token, and represent each suspicious flow as a sequence of tokens, then use Smith-Waterman algorithm extracts signature of a type of polymorphic worm. This method makes up the deficiencies of losing some important short sequence which Polygraph system may takes.(4) The feasibility and effectiveness of the model is verified by experiments...