A Method Based On Instruction Characteristics During Program Operation For Detecting Ropattack

With the rapid development of computer technology,offensive and defensive tech-nologies related to computer security are constantly evolving.This thesis focuses on ROP attacks(Return-oriented Programming)among many attack techniques.The attacker con-trols the return address of the source program to indirectly manipulate the direction of the program to perform a specific attack(usually to obtain control permissions).At present,ROP attacks usually use short-instruction tool chains(gadgets)in the static and dynamic libraries in the system.By splicing these gadgets,they can complete specific target at-tacks.The detection and defense methods for it have also undergone a long period of evolution: the initial static control flow has gradually developed to the current hardware detection method.Although the current popular hardware detection has the advantage of fast detection speed,its accuracy is low,and it will be affected by the system envi-ronment.Based on the shortcomings of low accuracy of hardware detection,this thesis designs a KVM-based ROP attack detection system based on the combination of software and hardware.The software method refers to the use of dynamic instrumentation to ana-lyze instructions,and the hardware method refers to the detection method based on LBR(Last Branch Record).The main contents of this thesis are as follows:First,by analyzing the principle of ROP attack,the ROP attack process is restored.In the restoration process,according to the inherent attributes and characteristics of the ROP attack,the three instruction characteristics of call,ret,and jmp are screened: for example,when the detection system is executed at a certain point in time or a function,whether the jmp instruction is inter-function Whether the number of jump,call and ret instructions is inconsistent,etc.Based on the abnormal characteristics of these instructions when they are attacked,a complete set of dynamic detection schemes for different instructions has been developed.Secondly,through the comparison of the two virtual machine architectures of KVM and XEN,it is found that KVM has the advantages of good scalability and open source,which is convenient for system development and upgrade.In addition,the interface be-tween the virtual machine and the host to exchange data in KVM is easy to call,and the code is simple to implement.Based on the above advantages of the KVM architecture,it is determined to use the KVM architecture for development to ensure the safety of the detection system.Thirdly,the new detection system innovatively proposes a funnel method to filter programs.In the first layer,the LBR detection module is used to detect most of the ROP programs,and then the dynamic instruction detection module is used for the ”normal pro-grams” that cannot be detected by the LBR.This module performs in-depth inspections,and finally obtains accurate inspection results.Through this nested detection method,the suspicious program is actually classified according to the probability of determining a ROP attack,and the detection accuracy of the entire system is improved.Finally,in the experimental part of this thesis,an independently constructed attack sample set and CVE buffer overflow vulnerability samples are used to evaluate the accu-racy of the detection system.Experimental results show that,in the face of original ROP attacks and variant ROP attacks,the system can maintain rate of detection accuracy more than 90%,which is significantly higher than the previous hardware detection accuracy.