Research On Defense Technology Of Buffer Overflow Attack Based On Exploits Detection

Buffer overflow attack belongs to the most serious threats to the Internet. By exploiting buffer overflow vulnerabilities in the remote process which provides important network services, skilled intruders can inject malicious packets with carefully crafted content that overflow a static buffer in the victim process. This allows the intruders to alter the execution flow of the service to execute malicious code. Since it is the first and most important step to inject malicious code to target process successfully to implement an attack, it is very meaningful to research on the detection of buffer overflow malicious code.This thesis summarizes the deficiency of contemporary methods for defending the existing buffer overflow attacks and introduces the principle and major technology of buffer overflow attacks. In this thesis a network-level polymorphic shellcode detection scheme based on emulation is introduced as a basic scheme with its feasibility, rationale and limitations. In order to improve the accuracy and speed of the basic scheme, this thesis proposes an improved detection scheme based on the execution behavior of polymorphic shellcode. As the basis of the detection algorithm, a behavior model is put forward by deeply researching on the behavior signature of polymorphic shellcode. This model could distinguish the execution behavior of polymorphic shellcode from the random data more accurately. Furthermore, a detection algorithm based on the behavior model is designed and realized. For mitigating the execution overhead of detection system, the optimization of using stream pretreatment and simulation loop optimizing is proposed. At the same time, this thesis also presents a detailed description on the key techniques for the implementation of the detection scheme and performance optimization.The detection scheme is developed with C language and implemented on Linux operation system and "libemu" which aims at creating a generic x86 emulation API as an independent library. In the end, the false negative of the detection scheme is tested by using polymorphic shellcode samples produced by six polymorphic shellcode engines which provided by MSF. The false positive and system performance is tested and analyzed by using the dataset gathered from real network environment. Compared with the basic scheme, the experiment result shows that the proposed detection scheme outperforms the basic scheme in both speed and reliability.