Rootkits and related attacks prevention and detection

Rootkit technology has evolved tremendously in the past two decades. Instead of launching the old-fashioned one-time worm/virus attacks, attackers nowadays prefer leaving a backdoor for easy returning exploits. In the meantime, rootkits have become even more popular as intrusion detection technologies advance. Stealth becomes an urgent factor for the attackers to succeed. The main functionalities of rootkits are to provide stealth. With the help of rootkits, the attackers can silently modify components without being detected. They can exfiltrate confidential information through covert channels from a higher to a lower security level without being identified. The attackers can even hide their traces by erasing the log files post-attack.;Due to the nature of operating system design, user-space rootkits are not fatal threats to security vendors any more since they always have lower privilege than the kernel-space applications. As the rootkits move to the kernel space, the arms-race between the attacker and the intrusion detection systems is in endless competition. Both parties have same privilege level and both parties can overwrite each other. Whoever gets there first wins the game. Among all the techniques used by rootkits, kernel-space hooking has become the most popular method. Kernel hooking is achieved by modifying a file/address pointer to alter the original execution path of a program. Thus, kernel-space hooking is relatively easy to achieve from the attacker's aspect. In the meantime, kernel-space hooking technology is powerful as it can even modify the functionality of system calls - the fundamental services between the application and the kernel. However, kernel-space hooking has become so popular that not only attackers but also anti-malware vendors are using this technology, to prevent/detect attacks. This causes a high positive rate of rootkit kernel hooking detection which makes it urgent to distinguish innocuous rootkits from malicious ones. In this dissertation, a formal hypothesis mathematical model is proposed to categorize rootkits into malicious or innocuous class. This dissertation also analyzes rootkits and their related attacks from both system and network perspectives. It analyzes covert channel rootkits and implements a prototype to prevent a covert storage channel rootkit from exfiltrating confidential information between attackers by introducing noises. The IP spoofing detection appoach prevents attackers from achieving their first step of escaping from detection accurately. If the attacker is able to bypass network prevention/detection, our scheme is able to provide further prevention/detection in the hosts. User-space rootkits can be prevented by applying kernel space SSDT (System Service Descriptor Table) permutation by introducing the concept of diversity. We gain higher privilege and tamper proof the kernel by using virtual machine introspection to monitor kernel-space hooking rootkits. Overall this dissertation provides a multi-level rootkit prevention/detection infrastructure that produces promising results.