Research On Security Of Virtual Machine System In Cloud Computing

Cloud computing platform provides services to users through Internet. This open modenot only facilitates the access by users, but also brings potential security risks. Usually, cloudcomputing platform utilizes virtual machine system as its underlying architecture. Conse-quently, the security of virtual machine system is of paramount importance to the security ofcloud computing. According to this background, this thesis studies three aspects of securityof virtual machine system in cloud computing.In cloud computing platform, especially in private cloud and community cloud, thecommunications between virtual machines (VMs) are necessary. However, this interactionprovides a possible channel for the propagation of attacks and malicious softwares. For thisreason, a mechanism is needed to guarantee the security when virtual machine communicateswith each other. In this thesis, we propose Virt-BLP model, which is a mandatory accesscontrol (MAC) model tailored to virtual machine system. It well satisfies the requirement ofmulti-level security (MLS) in virtual machine system. A series of elements, security axioms,and state transition rules are defined in Virt-BLP model. In cloud computing platform, guestvirtual machines (guest VMs) are used to provide services to users, while privileged virtualmachine (privileged VM) and virtual machine monitor (VMM) are managed by cloud serviceprovider. According to this property, Virt-BLP model defines privileged VM as the trustedsubject when it acts as subject. Some state transition rules could only be enforced by trustedsubject. As a result, privileged VM can manage and control the communications betweenVMs. That is to say, Virt-BLP model supports MAC and partial discretionary access control(DAC). Based on Virt-BLP model, we design and implement a MAC framework applicableto MLS in Xen, which is called VMAC. The experimental results show that the functions ofVirt-BLP model are mapped into the VMAC framework successfully. Moreover, Virt-BLPmodel is a versatile model, based on which other virtual machine systems could establishtheir own MAC frameworks.As guest VMs provide services to users, their security is of significant importance toprovision of secure cloud computing services. This thesis studies the security of user levelapplications and OS kernel in guest VMs respectively. We propose an in-VM measuring framework called Hyperivm, to determine the status of user level applications in guest VMs.The measurement module (MM) measures running executables in guest VMs. All measure-ment values are transferred to privileged VM through inter-VMs communications mecha-nism, and are stored in measurement table (MT). Reference table (RT) containing the trustedmeasurement values of running executables is used for verifying the status of executables.The trusted platform module (TPM) is leveraged to guarantee the integrity of MT and RT.Moreover, we design a module called memory watcher (MW) to determine the status of MM.A working prototype of this in-VM measuring framework is implemented on paravirtualizedXen, which could guarantee the security of user level applications in DomU. Meanwhile, itshows good efciency in performance evaluation.Compared to the security of user level applications in guest VMs, the security of OSkernel in guest VMs is more important. For this reason, we propose a dynamic monitoringframework called Hyperchk to guarantee the runtime security of OS kernel in guest VMs.This framework is deployed in privileged VM, and monitors the kernel memory of guestVMs via VMM. For privileged VM and VMM are transparent to the outside, they run withhigh security. As a result, the process of retrieving kernel memory of guest VMs is security,and furthermore the monitoring results are reliable. The key values used in Hyperchk frame-work are retrieved by searching the kernel memory of guest VMs, which largely increasesthe robustness of monitoring process. Besides, Hyperchk framework adopts the schemeof self-adjusting monitoring frequency according to the runtime CPU load. This schemeincreases the detection rate, and also decreases unnecessary overhead. With customizablepolicy center, Hyperchk framework is scalable and flexible. Moreover, a working prototypeof Hyperchk framework is implemented on paravirtualized Xen. This prototype presents ef-fectiveness of detecting kernel rootkits, and just incurs acceptable overhead under diferentworkload conditions.