Buffer overflow is a common software flaw, which constitutes a severe threat to the safety of computers. The paper advances an approach that assists safety analysts to detect buffer overflow in the binary code. With the analysis of unsafe functions as its core, this approach is called A Buffer Overflow Detection Method Based on Unsafe Functions.Through analyzing the causes of buffer overflow vulnerability and the conditions under which buffer overflow can be exploited, the paper sums up the basic features of the buffer overflow that can be exploited and hereby proposes the buffer overflow detection method by analyzing and monitoring unsafe functions in the binary. Two key techniques of this method, namely, unsafe functions identifying technique and unsafe functions parameters monitoring and controlling technique, are expatiated subsequently. The paper proposes to identify the inline forms of unsafe functions by matching the key instruction sequence, to identify the unsafe functions of the Mov+Jxx kind by analyzing the characteristic and to intercept arbitrary codes in binaries by extending Detours. Based on the two techniques, the paper designs and implements the prototype tool for buffer overflow detection directed at the binary object code.The prototype tool is tested in the paper. The outcome indicates that the accuracy is 100% when the tool is identifying unsafe functions in the library, and that there are a few cases of misreport with regard to common unsafe functions. The impact of the prototype tool on the performance of the source programs when it is monitoring and controlling unsafe functions is determined by the amount of the programs and the frequency of these functions being called in the source programs. With the help of the tool, analysts have fewer codes to audit manual than before. A case study of the buffer overflow in Windows Movie Maker being detected using the method and the tool mentioned above is presented in the conclusion part of the paper. |