Using Hardware Isolated Execution Environments for Securing Systems

With the rapid proliferation of malware attacks on the Internet, malware detection and analysis play a critical role in crafting effective defenses. Advanced malware detection and analysis rely on virtualization and emulation technologies to introspect the malware in an isolated environment and analyze malicious activities by instrumenting code execution. Virtual Machine Introspection (VMI) systems have been widely adopted for malware detection and analysis. VMI systems use hypervisor technology to create an isolated execution environment for system introspection and to expose malicious activity. However, recent malware can detect the presence of virtualization or corrupt the hypervisor state and thus avoid detection and debugging.;In this thesis, I developed several systems using hardware isolated execution environments for attack detection, malware debugging, and sensitive operations. My research approach combines 1) the isolated execution concept with 2) hardware-assisted technologies. It leverages System Management Mode (SMM), a CPU mode in the x86 architecture, to transparently detect and debug armored malware and perform sensitive workloads. This research uses SMM to secure systems with a minimal Trust Computing Base (TCB) and low performance overhead. In addition, I develop a BIOS-assisted isolation environment that is capable of running a secure commodity OS.;To demonstrate the effectiveness of my research, several prototypes of using SMM as the isolated execution environment are implemented. First, I use SMM to introspect all layers of system software, including applications, OSes, hypervisors, and firmware. Secondly, my research leverages SMM to transparently debug armored malware and achieve a higher level of transparency than state-of-the-art systems. Lastly, this thesis uses SMM to securely perform password-logins without trusting the operating system and prevents ring 0 keyloggers.