Research Of Buffer Overflow Attack Detection Based On Data Unreliability

Since the Morris, the worm which firstly uses buffer overflow vulnerabilities to spread, breaks out in 1988, buffer overflow attack has been one of the most serious computer security attacks. In recent years, it has become the major attack method in advanced persistent threat (APT). The attackers use seven buffer overflow vulnerabilities in APT action against Iran’s nuclear facilities. Although industry and academia have proposed a bunch of techniques to protect software and operating system, the number of attacks has been in a steady growth. Also attacking techniques are developing from buffer overflow to recent Return-Oriented-Programming (ROP). Vulnerability attack detection is a hot area in information security, especially binary code-level detection is the focus in industry. However, the complexity of x86 instructions brings many challenges.Considering the feature that malicious code derived from the original data, which is received by process and is untrusty, this thesis proposed a detection method based on data unreliability. When the process modifies control flow, if the code which will be executed is a substring of the original data, it indicates that the attack is occurring. To avoid the huge overhead brought by frequent code match, this thesis proposed a pre-treated method. That is, the obviously legal target address and the suspicious will be distinguished. And only the code at suspicious target address will be matched with the original data. This method reduces the overhead effectively.Since the ROP chain is derived from the original data when ROP attack occurs, similarly, the original data is also untrusty. Hence, the idea of data unreliability can also use to detect ROP attack. And this thesis proposed a ROP attack detection method based on data unreliability. When a suspicious ROP chain is detected, the content in the stack pointed by the extended stack pointer (ESP) will be matched with the original data to determine whether the ROP attack is occurred or not.Depending on the dynamic binary instrumentation technique, this thesis implements a prototype buffer overflow attack detection system based on data unreliability under Win32 platform. This thesis uses the prototype system to detect a variety of buffer overflow attacks. The experiments show that the detection method based on data unreliability can achieve good results in accuracy. Meanwhile, this method also reduces the overhead to an acceptable range.