Secure Program Execution Through Hardware-Supported Isolatio

One of the challenges in securing today's computing systems is how to efficiently protect critical parts of security-sensitive applications from attacks that are launched using untrusted or compromised system software layers. Modern operating systems (OS) and virtualization layers are growing into large and very complex pieces of code. Due to their large size and complexity, it is virtually impossible to design them without exploitable vulnerabilities. Despite a plethora of protection techniques, many recent attacks that exploit vulnerabilities in systems code that bypass existing protections have been successfully demonstrated. Instead of attempting to prevent software bugs or mitigating all possible routes for their exploitation, a fundamentally different approach is to completely isolate security-sensitive parts of application code from potentially compromised system software layers. With significant interest from many hardware vendors including Intel, ARM, AMD and IBM, isolated execution has a potential to become a future standard of secure computing.;The first part of this dissertation work is dedicated to analyzing challenges in traditional systems and introducing Iso-X --- a flexible hardware-managed architecture for supporting isolated execution. Isolation in Iso-X is achieved by creating and dynamically managing compartments (isolated software modules) to host critical fragments of code and associated data. The proposed solution provides fine-grained isolation at the memory-page level, flexible allocation of memory, and a low-complexity and hardware-only trusted computing base. It requires minimal additional hardware, a small number of new ISA instructions to manage compartments, and minimal changes to the operating system.;Then, we examine some future challenges faced by isolated execution environments. In particular, we discuss the vulnerability of isolated systems to side-channel and covert channel attacks due to shared physical hardware resources. Then, demonstration of several such new attacks that apply to both traditional and isolated systems is presented along with description of possible mitigation strategies.