Design Of Anti-debugging Based On Hardware Virtualization Technology For Software Protection

Debugging is a method that usually facilitates the dynamic analysis of run-time application for software development. However, it is also a double-edge sword, as debugging could be adopted by malicious attackers. Nowadays software protection is increasingly significant, so is the relevant protection mechanism involving anti-debugging. Although we have various technologies and principles, the anti-debugging problem is still an important challenge because the traditional software anti-debugging owns no ability to utilize a higher privilege level than operating system kernel. In other word, malicious code can implement relative strategy to prevent anti-debugging operation from coming true. Fortunately, the Hardware Virtualization Technology can make VMM play on a more privileged position than OS, successfully breaking through traditional anti-debugging approach and providing bright prospects for us.This paper proposes a Light-Weight and Anti-debugging Framework Based on Hardware Virtualization Technology (LAHF). Hardware Virtualization Technology is based on chip-processors, which run directly on hardware, and make VMM a system-level virtual machine. It can not only provide security and isolation effectively, but control code of Ring 0 layer and above. To improve its expandability and universality, LAHF is designed with multi-layer and plenty of interfaces. Besides, LAHF is packaged as a driver, so it can be easily installed into or uninstalled from the X86 architecture and Windows XP (32-bit version) environment.The focus of the project is to study how to utilize Hardware Virtualization Technology to identify and monitor the debugger, which is the so-called hardware virtualization anti-debug technique in this thesis. The focus of the technique lies in identifying whether or not the behavior is a debugging process as well as whether or not the process would affect the running of the original program. Through an in-depth analysis of the mechanism of Windows debugging, and by making full use of the benefits that Hardware Virtualization Technology offers, we find a satisfactory anti-debugging approach and then implement it. Besides, on account of the narrow application of current anti-debug technique, the anti-debug framework proposed in the thesis can monitor various kinds of debuggers on Windows platform.Next, in order to prevent the discovery and attack by malicious code, we utilize the Hardware Virtualization Technology to design the self-hidden module for LAHF. The self-hidden technique consists of two parts, namely page table copy and page table fraud. Page table copy means that LAHF copies the page table content in kernel mode of the operating system to a set of private page tables it has established. While page table fraud means to deceive the operating system that it has found the actual physical address, which is actually not, by modifying the page table in kernel mode.Finally, some experiments are done to verify the hidden and anti-debugging ability as well as performance of LAHF. The results indicate that LAHF can anti-debugging multiple traditional debuggers, without being detected by debuggers and OS. Besides, the total overhead induced by LAHF is below 5% in terms of performance tests.In brief, this paper successfully provides LAHF, a tool that detects the behavior of debuggers as well as makes itself imperceptible to OS and debuggers. In this way, LAHF can timely find abnormal debugging behaviors, while users don't worry about malicious code. In addition, its distinct design facilitates further development.