| As virtualization technology has been widely used in the computing world, understanding its security properties and implications has become essential for leveraging it for security research. This dissertation studies virtualization technology from three aspects. First, we investigate the memory sharing mechanism used in current mainstream virtual machine monitors, and reveal its security implications; second, we study a reverse information retrieval problem in virtualized environments; third, we attempt to improve the virtual machine introspection technique, in particular, we propose to use user-level invariants, rather than kernel level information, to reconstruct key data structures of the guest OS.;Memory sharing, also named memory deduplication, has been widely used in various commodity hypervisors. While this technique improves memory efficiency, it has a large impact on system security. We investigate the security implication of memory deduplication from the perspectives of both attackers and defenders.;Virtual machine extrospection (VME), which we define as a procedure to retrieve hypervisor information from within a guest OS, is an open problem and has not yet been comprehensively studied before. In this dissertation, we take the initiative and study this reverse information retrieval problem. In particular, we investigate how to determine the host OS kernel version from within a guest OS. Building on our detection of hypervisor features and bugs, we present a novel framework called Hyperprobe that for the first time enables users in a guest OS to automatically detect the underlying host OS kernel version in a few minutes. We implement a prototype of Hyperprobe and evaluate its effectiveness in five real world clouds, as well as in a controlled testbed environment, all yielding promising results.;Virtual machine introspection (VMI) is an approach to inspecting and analyzing the software running inside a virtual machine from the hypervisor. The existing VMI tools rely on up-to-date kernel information of the target operating system (OS) to work properly, and this requirement prevents these tools from being widely deployed in real cloud environments. In this dissertation, we present a VMI tool called HyperLink that partially retrieves running process and module related information inside a virtual machine without source code. While current introspection solutions support only one or a limited number of kernel versions of the target OS, HyperLink is the first one-for-all introspection tool. We validate the efficacy of HyperLink under different versions of Linux, Windows, FreeBSD, and Mac OS X. We demonstrate that HyperLink can help users detect real-world kernel rootkits and play an important role in intrusion detection. Due to its version-agnostic property, HyperLink could become the first introspection and forensic tool that works well in cloud environments. |
