Research And Implementation Of Virtualization Security Mechanism

Virtualization technology provides the virtual hardware resources transparently to the users through simulating the underlying hardware resources of the computer. This could improve the utilization of the system resources, and make the system flexible and manageable. Therefore, virtualization is being widely used in today’s various systems including clouds. Compared to type-Ⅰ (bare-metal) hypervisors, type-Ⅱ (hosted) hypervisors could directly benefit from the host OS, and are much easier to install and maintain, so they are increasingly being adopted in virtualization systems. Because of its high privilege and vulnerability, the hypervisor probably becomes the target of malicious attacks. Once the hypervisor has been compromised, it could probably be used to snoop or undermine the host OS and the guests, which is a serious threat to the security of the virtualization environment.The existing studies towards the security of the virtualization environment are mostly focused on the consolidation and enhancement to the hypervisor. Whereas removing all the bugs from the hypervisor code is too difficult to achieve, because of the large code base and complex internal logic of the hypervisor. Therefore, in this dissertation, we assume the hypervisor is untrusted. We have designed a system named Hyper-Guard which can effectively isolate a vulnerable type-Ⅱ hypervisor from compromising the host OS or the gests, and have implemented a prototype of Hyper-Guard for the popular open-source type-Ⅱ hypervisor KVM. The contributions and innovations of this dissertation are as follows:First, we have studied and analyzed the reasons cause security issues, the threats in the virtualization environment and the approaches to improve the virtualization safety in depth.Second, we have designed and implemented a system named Hyper-Guard that can sternly isolate type-Ⅱ hypervisors. It is the first safeguarding system specially designed for type-Ⅱ hypervisor architecture. Hyper-Guard consists of two key components, i.e., hypervisor isolation runtime environment and Hyper-Guard controller. In these two components we have innovatively proposed and implemented some isolation-related mechanisms and technologies, such as the access control of memory and instructions and hypervisor reflection etc. We also apply instruction alignment and system call interposition mechanism to our solution. With the cooperation among these mechanisms and technologies, the hypervisor is well isolated.Third, through the security analysis and the performance evaluation, we have demonstrated that Hyper-Guard can effectively secure the virtualization system as we expect with a low performance overhead (less than5%).