Research On Key Technologies For Software Security Protection Based On System Virtualization

In recent years, with Cloud Computing techonology sweeping the globe, system virtualization technology, as the basic techonology of Cloud Computing has been widely used. However,traditional security problems still exist and become even worse as the development and populatity of Cloud Computing. Software is the soul of information infrastructure. The runtime behavior of a software may not as trustable as user or designer expected either because internel shortcoming of software design or serious attacks from outside, which could induce huge ecnomic damage both for individual and for the socieoty. Security of software as foundmental problem of other security issues, has and will always be key problem of information security.There're fatal defects in traditional secutity systems, such as sandboxing system, anti-virus tools,when deployed in cloud enviroment--they run in the same address space with the softwares we want to protect and so do the Operating System. Consequently, they face same attacks as target softwares and security mechanisms they used could be circumvented and even closed once and for all. Meanwhile, it's hard to manage and deploy traditional security tools in virtulization platform on which a huge amount of virtual machine are running symoutaniously. System virtualiztion draws attention from both industry and acdamic fields because of its isolation and higest previledge properties. Software security protection could generally be divided into two complementary aspects, one of which is the prevention of compromising other softwares on system software stack by target software themselves, the other is to protect data and runtime excution of target software from attack launched by untrusted OS or preveliged software. Using user defined security policy as reference to respond to monitored abnormal action.In order not to affect normal execution of the non target software; the effective monitoring points in the client virtual machine system are limited to the target software process context using shadow memory mechanism. A domain specific language of virtual machine introspection is designed based on in-depth study of the foundmental technology of VMI. Using the language, security application of VMI could be developed and managed in an efficient, natural and intuitive way. Based on the above four key technologies, the research and development of the system are carried out, and some achievements are obtained:(1) In order to find a way to unify all foundmental VMI technology in one formalism to use VMI to obtain a broad spectrum of information from a virtual machine for security purposes, a domain specific language of VMI named VmiDsl was designed. From perspective of programming language, the abstraction level of VmiDsl is higher than API so that security application of VMI could be developed and managed in an efficient, natural and intuitive way using VmiDsl.In order to implement the prototype, payparsing library of Python was used to build syntax parser of VmiDsl, and the underlying implementation was fulfilled in KVM. Two use cases implemented by VmiDsl are tested and anylised.(2)An application sandbox system VxBox based on system virtualization was proposed to control user space runtime behavior of target program according to customised security policies out-of-the-vm. The security policies were managed by the management virtual domain.Even if privileged software hijacked in the the target virtual machine, VxBox cannot be bypassed, compromised or closed, or tampered with its security policy. A prototype system of VxBox was implemented in the x86 and AMD64 hardware platform respectively based on para-virtualization Xen. The experiments demonstrated that a wide variety of applications software (such as a web server or a virus scanning tools, etc.) could be controled by VxBox system with higer efficiency than using ptrace to monitor target software inside target virtual machine;(3) An "out-of-the-vm" security system to protect target software memory and related virtual disk data called VxWall is proposed. Shadow memory technology based on. VxWall extends memory management from VMM to provide target program different memory view for the different operating modesrespectively.When target processes running in user mode, the memory access is through the original "real pages",and when the kernel represents the process to excute in kernel mode, the memory access is through dummy physical memory pages.This shadow memory mechanism ensures that even if the target virtual machine being hijacked,it could be very hard to induce the target software memory data leakage. For protection of target software related virtual disk data, VxWall using a trusted virtual management domain to manage these documents, while dummy files stored in target virtual machine just for consistency. When the target software invoked file operation system calls, the VMM the core module Vx-core intercept those system calls and simulate respective operation, as a middleman between the virtual management virtual domain and the target domain for data file content exchange. Prototype system is implemented on the AMD64 platform and the effectiveness and performance analysis are carried out.(4) A security system called VkXeck for controlling target software kernel behavior was proposed. Kernel control flow compliance of target software was monitored by VkXeck according to user specified security policy ithin the context of the target process through multiplexing kernel address space memory pages and changing corresponding page table entries of targeted program.VkXeck solved the problem of current method that caused a serious decline in system performance for whole kernel hooking and without strictly limit the kernel extension mechanism. The prototype system is implemented in xen-AMD64 and the effectiveness and efficiency are analyzed.