ROP Attack Detection Based On Virtual Machine Introspection

A major challenge for today's computer software security protection mainly focused on memory-related overflow attacks.In order to bypass classic defense mechanisms such as "write xor execute",intruders often choose code reuse to achieve attack intent.Return Oriented Programming(ROP)is a code reuse technique that allows intruders to hijack a program's control flow and perform arbitrary malicious behaviors.ROP attacks are designed to build malicious code by linking a short instruction sequence ending with a return instruction,thereby threatening application modules in user space and even kernel space.The current detection methods for ROP attacks are mostly under a single machine.The detection system itself has certain security risks.The virtual machine introspection technology allows the detection system to transparently monitor the client virtual machine user process in the highly isolated virtual machine management layer.It also improved the robustness of the detection process.This thesis analyzes the core gadget chain characteristics by restoring the details of ROP attacks,and proposes a hardware-assisted multilayer detection method for ROP attacks.This method monitors the instruction execution process of the user process of the virtual machine in real time,checks the legitimacy of the returned branch through the branch records in the Last Branch Record(LBR),and detects the ROP attack through the threshold determination of the gadget chain length.For long gadgets that can bypass the chain length threshold determination,the ROP attack is further identified by the number of branch instructions,mispredicted branch instructions,and instructions threshold for a gadget chain in the Hardware Performance Counter(HPC).Based on the above algorithm,and taking the KVM virtualization framework as the background,this thesis implemented ROP attack detection prototype system VMIROP using virtual machine introspection technology.It mainly combines the functions of the KVM kernel module to capture user space system calls in the guest virtual machine,and restores process switching in the virtual machine by maintaining a CR3 register lookup table.Under the premise of identifying the target user process,the perf kernel module obtains the underlying hardware information,and finally executes hardware-assisted ROP detection logic.After actual offensive and defensive experiments,the detection rate of the prototype system on the construction samples reached more than 90%,whichcan effectively identify ROP attacks caused by memory overflow vulnerabilities such as CVE-2017-9430.At the same time,the average processor performance loss in virtual machine was 6.4%.