Dynamic Detection Of Buffer Overflow Vulnerabilities In Binary Environment

Buffer overflows are security vulnerability that widely exist in all types of operating systems and programs, but also most frequently used by hackers. At present the majority of the buffer-overflow detection tools are based on the source code, this greatly limits the use. There are some inherent flaws in the traditional static detection of buffer overflows in binary environment, mainly in the type of pointer aliasing, as well as identification variables. This paper proposed a new way to find buffer overflow vulnerabilities in binary environment combined dynamic and static analysis that make up for deficiencies in the majority.After analyzed x86 CPU architecture and instruction, it detailed binary files disassembling, and the design of virtual execution environment (VM). The VM has the same basic IA-32 execution environment as a real CPU that can run native x86 instructions. By visiting the state of instruction execution, tracking the changes in register, it solved the problem of pointer aliasing via VM. It records each instruction address that accesses the memory, than calculates the variables addresses in the final statistics, resolved the problem of the precise variables identification.Thare is a large number of library functions in the modern programs. In order to avoid these library functions as much as possible, this paper presents a new fast library function recognition technology. Compared with the existing technology, it improved the recognition accuracy, but the identification database's storage and reconstruction is complicated.By the interval graph theory graph theory, the program flow diagram is divided into disjoint subgraph of the interval in the control-flow analysis. Through simpling interval graph, it identifies the nested control structure in the flow diagram step by step. In the data flow analysis, it calculates the traditional data flow equations, and puts forward a simple and effective type of recognition algorithm for the binary. For the 2 string copying operations, i.e. insecure function invokes and copying memory loops, it researches buffer overflow caused by the later and present a flow sensitive and context-insensitive algorithm to detect it. In identification of the key property in the loop, once again via VM, it calculates the difference between loop statuses, finds changes in volume, obtains the control variables and control displacement.The buffer overflow detection model in practical applications is introduced. Experimental results show that the model can detect common buffer overflow vulnerabilities well.