Hardware-assisted Isolation Mechanism For Intra-VM Execution Environment

The operating system is one of the most important components in the whole system,on top of which runs various applications ranging from desktop softwares to servers.The OS assures the stability,efficiency,and security of these applications.Most of servers run a macro kernel inside,such as Linux,which shares the kernel address spaces among all processes.Such a design makes the communication cost between different subsystems relatively small but increases its security risks.The huge codebase makes the kernel bug-prone,while the lack of isolation makes it more vulnerable,especially considering that kernel owns a higher privilege than most of applications.Once a single part of the kernel component is broken,the whole system will be fully exposed to security threats.Therefore,enhancing the isolation of operating system,especially the kernel,has been a hot research field over the years.The hardware-assisted virtualization mechanism has shed new light on the issue above.The Extended Page Table(EPT)and other features makes the isolation between different address spaces more effective.Unfortunately,most of the current works suffer from the performance overhead of switching EPTs by trapping into hypervisor,a few works utilize VMFUNC instruction to reduce such cost,but do not solve the security issues of switching between multiple mutually distrusted execution environments.Besides,most of the works fails to eliminate the semantic issue between hypervisor and virtual machine when performing checks on suspicious memory accesses.Regarding to the issues above,this thesis designs and implements an intra-kernel hardware-based isolated execution environment mechanism with the help of EPT,VMFUNC instruction,and Virtualization Exception feature.Below are the main contributions.The isolation mechanism leverages Extended Page Table(EPT)feature to enforce address space isolation between execution environments,so that an attacker could not read or modify the unauthorized data in other environments.The mechanism leverages VMFUNC instruction to accelerate the EPT switching procedure,which greatly reduces the cost of switching EPT,causing less than 2% overhead to Apache and Memcached program,and other programs during the running or the system.The isolation mechanism utilizes Virtualization Exception(VE)mechanism to check suspicious memory access,which enables the virtual machine to capture the suspicious memory access inside virtual machine without trapping into hypervisor.The virtual machine benefits from the full semantics inside it,which makes the check effective without suffering from the semantic gap.