Research On ROP Attack Mitigation Technology Based On Windows Platform

As the 21 st century's most dynamic and creative industry, the Internet is deeply rooted in every corner of human society. People enjoy the convenience brought by the Internet while do not take the associated security issues seriously. Network security events emerge endlessly in recent years, such as Google aurora borealis APT events, Hacking Team information leak and Baidu application Wormhole loopholes events. All of them have caused great damage to people, from which this serious situation was clearly indicated. In security incident, ROP attack attracted hackers' and security researchers' increasing attention since it is able to circumvent the most current defensive measures. This thesis was in view of the current vulnerability ROP in the offensive and defensive attack of severe situation, and based on the study of the normal process to jump. At last a new method of ROP attack mitigation was proposed, and the design implementation of corresponding mitigation prototype system was fulfilled as well.First, based on the principles of the debugger, hot dynamic link library analysis solution was proposed through the Windows system unique page guard exception, by which the HMAT instruction analysis tool was realized. The HMAT instruction analysis tool is able to detailedly analyze program execution characteristics of assembly instruction level during operation, and separately analyze the process of particular dynamic link library according to user input. Taking advantage of HMAT instruction analysis tool, this thesis displayed the analysis of the several representatives in the Windows system software, and then explained a detailed study on indirect jump instructions including call, jmp and rets. As a result, three types of indirect jump instructions in normal execution process were different from how they process when subject to ROP attacks. Therefore the three indirect jump instructions can behave as indicators to detect ROP attacks.Second, the next work was focus on the three essential issues in the process of the detection and prevention of ROP attacks, including when the detection should be performed in the course of a program's running, where the detection will work and by which method to recognize ROP attacks. The traditional detection method was also used for this part of work. As a conclusion, the ROP attacks can be clearly detected by the process of the key function was called, and the function selection criteria were defined as well. Therefore, this work innovatively proposed that the single-step debugging technique could be used to identify each instruction executed, and the identification for the completeness of indirect jump instruction sequence could be applied to determine whether the ROP is currently being attacked.Finally, a new method of ROP attack mitigation was proposed, and based on which a prototype system was implemented. In this paper, mitigation goals, and module design of the system were described in detail. Besides, detailed testing of ROP attack mitigation prototype system was carried out when using three vulnerabilities samples, and the results indicated that the system can effectively defend against ROP attacks.