Automated detection and containment of stealth attacks on the operating system kernel

The operating system kernel serves as the root of trust for all applications running on the computer system. A compromised system can be exploited by remote attackers stealthily, such as exfiltration of sensitive information, wasteful usage of the system's resources, or involving the system in malicious activities without the user's knowledge or permission. The lack of appropriate detection tools allows such systems to stealthily lie within the attackers realm for indefinite periods of time.;Stealth attacks on the kernel are carried out by malware commonly known as rootkits. The goal of the rootkit is to conceal the presence of the attacker on the victim system. Conventionally, kernel rootkits modified the kernel to achieve stealth, while most functionality was provided by accompanying user space programs. The newer kernel rootkits achieve the malice and stealth solely by modifying kernel data. This dissertation explores the threat posed by both types of kernel rootkits and proposes novel automated techniques for their detection and containment.;Our first contribution is an automated containment technique built using the virtualization architecture. This technique counters the ongoing damage done to the system by the conventional kernel rootkits. It is well suited for attacks that employ kernel or user mode stealth but provide most of the malicious functionality as user space programs.;Our second contribution is to identify a new class of stealth attacks on the kernel, which do not exhibit explicit hiding behavior but are stealthy by design. They achieve their malicious objectives by solely modifying data within the kernel. These attacks demonstrate that the threat posed to kernel data is systemic requiring comprehensive protection.;Our final contribution is a novel automated technique that can be used for detection of such stealth data-centric attacks. The key idea behind this technique is to automatically identify and extract invariants exhibited by kernel data structures during a training phase. These invariants are used as specifications of data structure integrity and are enforced during runtime. Our technique could successfully detect all rootkits that were publicly available. It could also detect more recent stealth attacks developed by us or proposed by other recent research literature.