Improving operating systems security: Two case studies

Malicious attacks on computer systems attempt to obtain and maintain illicit control over the victim system. To obtain unauthorized access, they often exploit vulnerabilities in the victim system, and to maintain illicit control, they apply various hiding techniques to remain stealthy. In this dissertation, we discuss and present solutions for two classes of security problems: TOCTTOU (time-of-check-to-time-of-use) and K-Queue. TOCTTOU is a vulnerability that can be exploited to obtain unauthorized root access, and K-Queue is a hiding technique that can be used to maintain stealthy control of the victim kernel.;The first security problem is TOCTTOU, a race condition in Unix-style file systems in which an attacker exploits a small timing gap between a file system call that checks a condition and a use kernel call that depends on the condition. TOCTTOU vulnerabilities are widespread and cause serious consequences. For example, according to US-CERT (United States Computer Emergency Readiness Team), such vulnerabilities exist in a wide range of applications, affect many operating systems, and often give the attacker unauthorized root access. Our research contributions on TOCTTOU include: (1) A model that enumerates the complete set of potential TOCTTOU vulnerabilities (e.g., 224 TOCTTOU pairs in Linux); (2) A set of tools that detect TOCTTOU vulnerabilities in Linux applications such as vi, gedit, and rpm; (3) A theoretical as well as an experimental evaluation of security risks that shows that TOCTTOU vulnerabilities can no longer be considered "low risk" given the wide-scale deployment of multiprocessors; (4) An event-driven protection mechanism and its implementation in the Linux kernel that defend Linux applications against TOCTTOU attacks at low performance overhead.;The second security problem addressed in this dissertation is kernel queue or K-Queue, which represents a new hiding technique that can be used by the attacker to maintain stealthy control of the victim system after a successful break-in. K-Queue-driven attacks can achieve continual malicious function execution without persistently changing either kernel code or data (from the "gold" distribution), which prevents state-of-the-art kernel integrity monitors such as CFI and SBCFI from detecting them. We have studied a concrete instance of K-Queue-driven attacks that use the soft timer mechanism found in nearly all full-featured operating systems. We demonstrate that an attacker can use soft timer interrupt requests (STIRs) to perform powerful attacks, including key logging, denial of service, and hidden process scheduling. To defend against soft-timer-driven kernel control flow attacks, we propose and implement an approach based on an automated static analysis of the entire kernel that identifies and catalogs all legitimate STIRs in a database. At runtime, a reference monitor in a trusted virtual machine compares each pending STIR with STIRs in the database, allowing the execution of only known good STIRs. Our defensive technique effectively mitigates soft-timer-driven attacks at a low cost (less than 7% for each of our benchmarks).;As the finishing touch of this dissertation, we design and implement a solution to the general class of K-Queue-driven attacks which can exploit IRQ action queues, tasklet queues, soft timer queues, and work queues. Our first contribution is a unified static analysis framework and a set of tools that can generate specifications of K-Queue summary signatures and the corresponding checking code in an automated way. We also design and implement a unified runtime reference monitor based on virtualization that validates K-Queue invariants and guards such invariants against tampering. Finally, we perform a comprehensive experimental evaluation of the scalability of our static analysis framework and tool set, which shows that different K-Queue analyzers have significant overlapping that can be exploited for better efficiency; and we carry out an evaluation of the complexity and runtime overhead of our K-Queue Checker which suggests ways for further optimization.