Virtual machine based mechanisms and tools for cyber attack prevention, analysis, and recovery

Throughout the last decade we have witnessed a widespread use of the Internet and a dramatic change in the way people communicate, do business, and present themselves to the world. It did not take long before criminals started exploring this rich environment seeking fun, pride and later illicit money and even war. In the light of this new generation of malware and attacker's motivations, complete defense strategies must address prevention, detection and response to attacks. In spite of that, the majority of efforts in malware defense currently focus on detection.;This dissertation addresses prevention and post attack analysis and recovery in the context of virtual machine (VM) environments. It provides a study of full system replay for post-attacks analysis where the execution of an entire system from a checkpoint can be faithfully replayed with low performance/space overhead. Building on this research, it describes the application of this replay approach on post-attack recovery from control-flow hijacking Internet worms. Finally, this dissertation challenges the traditional VM usage model that advocates placing security mechanisms only in the VM layer, letting the guest operating system (OS) run unaware of virtualization. It shows how collaboration between the guest OS and a VM helps bridge the semantic gap between these layers and provides stronger system protection. The dissertation additionally reports on implementations and proot-of-concept prototypes of these mechanisms, showing them to be effective for their respective scope. The implementations and prototypes validate our proposed approaches and have no false positives or negatives (in the context of prevention and for all the attacks used in our experiments), low performance/space overhead (post-attack analysis) and address zero-day attacks (post attack recovery).