Study On System Security Techniques In An Untrusted Kernel Environment

In commodity operating systems, the kernel is executed on the highest privilege level to manage the underlying hardware resources and provide applications with se-cure and isolated resource access interfaces. The kernel is the trusted computing base (TCB) of the entire system. However, the kernel has a huge code base, complex da-ta structures, widespread attack window, and is usually written in unsafe languages; more and more security vulnerabilities reports indicate that the kernel still has many bugs and errors and is not completely secure. Once the attackers compromise the k-ernel, they can acquire the highest privilege of the system and conduct any malicious operations that they want, including manipulating the underlying hardware, executing arbitrary code in system, read and write arbitrary data in memory or on disk.To solve this problem, previous approaches usually introduce a new TCB (e.g., virtual machine monitor) that runs at a higher privilege level than the kernel, and de-ploy and realize security protection mechanisms based on the newly introduced TCB. However, the frequent privilege level transitions between the TCB and the kernel in these approaches cause relatively high performance overhead. We address this limi-tation, we propose a same-privilege-level TCB approach. Our approach does not rely on a higher privilege level. Instead, we introduce a new TCB that runs at the same privilege level with the kernel, and deploy security protection mechanisms and pre-vent kernel-level attacks based on it. We demonstrate that our approach can achieve the same level of security as the previous approaches that rely on a higher privilege level. At the same time, since the switching between the kernel and the TCB in our ap-proach requires no privilege level transitions, the performance overhead is significantly reduced.The main contents of this paper are as follows:We present a method to achieve same-privilege-level TCB based on hardware virtualization. In particular, we make use of hardware virtualization mechanisms to intercept and validate kernel’s privileged operations and provide a secure execution environment for applications. Based on this, we successfully protect security-sensitive applications from the untrusted kernel.We present a method to achieve same-privilege-level TCB based on instruction address size. In particular, we modify the instruction address size of the instructions in kernel code to restrict the kernel’s memory accesses in the address space. With the combination of kernel code integrity and kernel control flow integrity, we successfully protect security-sensitive applications from the untrusted kernel.We present a method to achieve same-privilege-level TCB based on the combina-tion of SFI and address space isolation. In particular, we combine code sandboxing and address space isolation to intercept and validate kernel’s privileged operations and en-sure the secure isolation and trusted execution of the same-privilege-level TCB. Based on this, we realize active kernel monitoring.We present a method to achieve same-privilege-level TCB based on x86’s hard-ware mechanisms. In particular, we leverage the WP and NXE mechanisms provided by x86 to achieve the interception and validation of kernel’s privileged operations as well as to prevent the untrusted kernel from compromising same-privilege-level TCB. Based on this, we realize active kernel monitoring.We systematically perform security analysis and performance analysis on the above four proposed methods of same-privilege-level TCB. The analysis results demon-strate that our methods can provide the same level of security as previous approaches relying on a higher privilege level. In the meanwhile, the performance of our methods are much better than previous approaches.