Research Of Vulnerability Attack Detection On Binary Code

Network attacks have caused great threats since the Morris worm in1980’, and theroot cause of various attacks is software vulnerability. The number of attacks has beenin steady growth, though industry and academia have proposed a bunch of techniquesto protect software and operating system security. Attacking techniques have been de-veloping, from buffer overflow to recent Return-Oriented-Programming(ROP). Vulne-rability attack detection is a hot area in information security, especially binarycode-level detection is the focus in industry, however, the complexity of x86instruc-tions and WYSINWYX(What You See Is Not What You Execute) give rise to manychallenges.Binary code level detection can not only defend attacks, but also provide sugges-tions for vulnerability analysis and zero-day vulnerability mining. It is also beneficialto design effective compiler-and OS-level defense. However, current detectionsagainst attacks have several vital limitations. First, a large fraction of defense tech-niques require source code which is unavailable for commercial software, and thus bi-nary code analysis is a must; second, most detection techniques are pure static or dy-namic analysis that sacrifices accuracy or efficiency, and there is no seamless combi-nation. Particularly, for recent ROP attacks, there is no effective defense. In brief, cur-rent attacks demand new defense techniques, but classical defenses such as stack ca-nary, NX and signature matching are unable to meet the demands.This dissertation performs in-depth and elaborate research in three categories, in-cluding dynamic taint analysis, control flow integrity and complex shellcode. Weachieve four major contributions:1. Proposed several novel techniques to strengthen dynamic taint analysis, whichis able to accurately detect a major attack form-memory corruption attack. Designedand implemented efficient shadow memory, which can store and query shadow bit inlow time and space complexity. Proposed structured and hierarchy analysis of x86in-structions, which overcomes the difficulty of x86instruction complexity. Using suchinstruction analysis, we design accurate taint propagation policies. Traditional taint analysis propagate taint value only on instruction level, lacking of procedure levelpropagation. Two techniques are proposed to enhance effectiveness and accuracy ofprocedure-level propagation that they are remaining taint data clear and function sum-mary. The prototype system is developed using dynamic binary instrumentation tool.The system is evaluated with real-world attacks and experiment results demonstratethat our system can effectively detection memory corruption attacks, moreover, systemperformance has also improved.2. A novel technique against ROP is proposed. Current approaches rely on partialfeatures of ROP for detection, but they are unable to accurately distinguish normal in-struction execution and ROP attacks, which may cause high false positives. Throughreal ROP exploits study, we establish the manifested feature and functional feature ofROP and by using them to detect ROP attacks. Meanwhile, pure dynamic analysis in-curs high overhead, and more important, it cannot block attack before gadgets are ex-ecuted. We propose a new technique, black-box gadget emulation, to overcome theseproblems. The prototype system is built with dynamic binary instrumentation. ROPexploits are generated with an automatic tool. Experiment results show our system issuperior to existing ROP detection techniques in terms of both accuracy and efficiency.3. A new function pointer defense technique called FPGuard is proposed. FPGuardworks by sanitizing address that checks if jump targets fall in legal function addressspace. The major difference between FPGuard and CFI is that the checking code ofFPGuard is placed out of program, which can analyze indirect jumps. Besides tradi-tional code injection attacks, attackers are still able to mount ROP attacks. We alsofound there are non-entry function calls in C++programs. A two-step detection tech-nique is proposed. Tthe first step is coarse-grained range check, and the second step isfine-grained black-box gadget emulation. By evaluating real-world exploits, FPGuardcan accurately detection code injection and ROP attacks. System performance is guar-anteed, as FPGuard uses efficient data structure to store and query function addressinformation. FPGuard is the fist system that can detect both code injection and ROPattacks which are exploited by overwriting function pointer.4. A technique of detecting Self-Modify Polymorphic Shellcode(SMPS) is pro-posed. By studying real SMPS samples, we found a major characteristic of SMPS isdynamic code generation. Based on this feature, we propose an effective detection technique by combining static and dynamic analysis. First, it obtains static code bytesusing disassembling, and then obtains dynamic code bytes by emulation. If static anddynamic code are different, it implies a SMPS instance is identified. A simple and ef-fective filtering method is used to filter normal payload. We evaluate the system withSMPS instances generated by Metasploit, and results show it achieves satisfactory re-sults, and also by evaluating false positives with innocuous flows, it reveals false posi-tive rate is very low.