The Research On Defense Against Operating System Kernel Level Rootkits Via Virtualization Platform

With the development of information technology, the society has met an increasing demand of information security, which has become a serious problem nowadays. The security problem of the operating system could threaten the whole information system, because that the operating system is a fundamental component of the information system. As the main part of the operating system security, the kernel security may affect the safety of the operating system or even the information system. The attacks of kernel level rootkits are the major threats to the kernel of the operating system. Kernel level rootkits are able to control the operating system or hide their malicious behaviors by tampering with the kernel codes or data.The aim of this research is to find the defense method against kernel level rootkits. The basic idea of the method is to protect the integrity of kernel data. At first, it constructs the data-access graph and the function-call graph. Then it uses the two graphs as reference standard to protect stack and non-stack data in the kernel space. At last, it builds a safety protection model and an experimental prototype to defend against kernel level rootkits.The main contributions and innovations of this paper are as follows:1. A cross-platform algorithm is proposed to build the data-access graph and the function-call graph for the operating system kernel based on virtualization exceptions mechanism. The algorithm is not constrained by the software structures or the compiler rules, and it can obtain high precision and recall rates.In order to provide reference standard for the protection of stack and non-stack data in the kernel space, an algorithm for automatically building data-access graph and function-call graph is proposed. To build data-access graph, it records the write operations to the special memory via the page-fault exception mechanism of the hypervisor. As for the function-call graph, it acquires the invoking relationship from child functions to parent functions by intercepting the first instructions, call instructions and return instructions of memory functions, using the software breakpoint exception mechanism of the hypervisor. The experiment constructs the two kinds of graphs for x86 based 32-bit Windows XP, 32-bit Linux and x64 based 64-bit Windows 7 respectively. The results show that theprecision of data-access graph reaches 100% and the same with the function-call graph. Besides, the recall rate of function-call graph is 87%. The algorithm can support multiple x86/x64 based operating systems and is not constrained by the software structures or the compiler rules. The data-access graph and the function-call graph can be directly used as reference standard for the protection of stack data and non-stack data in the kernel space.2. A defense method for non-stack data in the kernel space is proposed. It uses the code segments of legal kernel modules, the data-access graph and the function-call graph as the trusted area to protect codes, heaps, data segments and BSS segments in the kernel space. It can defeat multiple types of rootkits with high reliability.In order to protect the non-stack data in the kernel space from MEP, KOH and DKOM rootkit attacks, a defense method based on trusted area for protecting non-stack data in the kernel space is proposed. At first, it builds trusted area using the code segments of legal kernel modules and monitors the discrete function pointers of the non-stack data to make sure they are in the trusted area. Then it constructs trusted area using the data-access graph and the function-call graph to make sure that the other kinds of non-stack data can only be modified by the instructions in the data-access graph, and the parent functions of these instructions are in the function-call graph. The experiment chooses 6 kinds of typical rootkits and builds 14 kinds of attack samples for 32-bit Windows XP to test the method. The results show that the method can defeat all kinds of the rootkits and attack samples. It can defend against the attacks of MEP, KOH and DKOM rootkits and prevent the page mapping attacks. The method can effectively protect non-stack data in the kernel space. Compared with the existing methods, the advantage of the proposed method is that it can prevent the DKOM rootkits from running. Furthermore, this kind of defense is more comprehensive and reliable.3. A defense method for stack data in the kernel space is proposed. It monitors the switching, replacing, creating and deleting process of kernel stacks to bind the execution unit with its kernel stack. It has a high level defensive ability and a large detection range, and it can protect all kinds of data on kernel stacks synchronously.In order to protect kernel stack data from the “return-to-schedule” and its extendedrootkits, a defense method based on binding execution unit is proposed. It monitors the switching, replacing, creating and deleting process of kernel stacks to synchronously change the read/write property of kernel stack memory. Thus, it merely allows the execution unit to write its own kernel stack and makes an execution unit bound with its kernel stack. Then it protects the relative code and data in the kernel space according to the data-access graph and the function-call graph. Thus, the execution unit cannot tamper with its own kernel stack data by executing malicious code. The experiment builds 6 kinds of attack samples for 32-bit Windows XP to test the method. The results show that the method can defeat all kinds of the attack samples, and defend against the “return-to-schedule” and its extended rootkits. It can protect all kinds of data on kernel stacks including return addresses, parameters and local variables.4. A cross-platform model against kernel level rootkits based on virtualization technology is proposed. Found on the model, an experimental prototype is implemented, which has strong defense capability and consumes a few resources.In order to protect the operating system kernel data from kernel level rootkit attacks, a defense model against kernel level rootkits is proposed. Based on the model, an experimental prototype is implemented. The prototype utilizes the defense methods for non-stack data and stack data to protect the memory data in the kernel space. Besides, it monitors the write operations to the key registers to guard the integrity of the data in the registers. In order to support multiple platforms, it recognizes the type of the operating system in the virtual machine, and then introspects and protects the semantic information. The experiment chooses 6 kinds of typical rootkits and builds 25 kinds of attack samples for 32-bit Windows XP to test the prototype. The results show that the prototype can defeat all kinds of the rootkits and attack samples. The performance cost is less than 3.1%. Moreover, it can also protect 64-bit Windows 7 and 32-bit Linux from rootkit attacks. Therefore, the prototype can effectively protect the kernel data of multiple operating systems with a small performance overhead.