Software Buffer Overflow Vulnerability Automated Excavation System

As malicious code wantonly spreads on the Internet as well as hacker attack events continue to rise, network security problems become increasingly serious. Software installed on computers is the root of network security problems. If software installed on computers is vulnerable, hackers and malicious code can easily penetrate and control computers via the software vulnerabilities. At present, no firewall or IDS can detect and react at this kind of attack based on 0Day (unpublished software vulnerability) because of their passive defense mode.Until now, software vulnerabilities are still the most important means of attacks employed by hackers. Nowadays, software vulnerabilities are the first and most important ring in the black-industry of the Internet. Among of all kinds of software vulnerabilities, buffer overflow vulnerability is the most serious one. According to statistics, almost 80% hacker attacks exploit buffer overflow vulnerabilities.If software vendors can detect vulnerabilities in their software as many as possible before software publication, the software will get more secure and the maintaining cost will get lower. However, the detection of vulnerability is a time-consuming process and relies on manual analysis heavily.To solve these problems, this paper introduces a novel method on the detection of buffer overflow vulnerabilities, which is highly automated. In detail, this method makes use of the combination of fuzzing technique and configuration files to generate malformed packets automatically. On this basis, dynamic debugging technique on Windows platform is used to interpret the exceptions during testing. Finally, all exceptions interpreted during testing are analysed automatically and an intelligent analysis report is given. Research work mentioned in this paper includes:1. Research the mechanism of buffer overflow, the ways to exploit buffer overflow vulnerabilities as well as the detection of buffer overflow vulnerabilities;2. Detail the design of Automated Software Buffer Overflow Vulnerability Discovering System based on the Windows operating system(Marmot prototype system); 3. Detail the design and implemention of the key modules of the prototype system, including script analysis module, dynamic debugging and vulnerable function tracking module, exception analysis module as well as the plug-in module for vulnerable function searching;4. Execute function test on Marmot prototype system. The test result has proved that the Marmot system reaches the expected goal.