Research On Technology Of Buffer Overflow Discovery Based On Hex-Rays

In all kinds of software vulnerabilities, the buffer overflow is not only one of the mostcommon security vulnerabilities, but also considered as the greatest threat to software security.Source-oriented vulnerability discovery method is widely studied, but the source code is not easyto obtain, and it cannot find the vulnerabilities introduced in compilation and link stage; thetarget for detection of binary-oriented vulnerability discovery is disassemble code, under thiscircumstance, the program structure is uncertain and data type information may get lost whencompared with source code, which could bring in trouble for vulnerability discovery.This paper researches the technology of binary-oriented vulnerability discovery based onthe technology of decompilation and decompiler of Hex-Rays, design and implementationvulnerabilities discovery prototype system. The accuracy of decompilation result is the basis ofthe thesis, this paper uses dynamic binary analysis method and function parameter matchingmethod to correct the decompiled results, aim to solve the problem of the inaccuracy of indirectcall destination code recognition and data type identification errors, extract the amendeddecompile results to construct Attribute–abstract syntax tree (A-AST) as a basic forvulnerability discovery. Also, the paper analyzes the manifestations of the buffer overflow in thedecompile results,builds the mode of library function call error and the mode of out of boundserror in copy-circulation, and then proposes the buffer overflow rothole positioning techniquebased on vulnerability mode. Besides the paper proposes the method of filtering the bufferoverflows based on data flow tracking and method based on the data association and the methodbased on reversed data flow tracking to reduce the false positive and improve the accuracy of theresults.This paper designs and implements a static binary-oriented buffer overflow vulnerabilitydiscovery system BODT, and has tested its functionality and performance. The test results showthat the system can greatly reduce the scope of the analysis in vulnerability discovery, andimprove the vulnerability discovery efficiency.