Radar: Compiler and architecture supported intrusion prevention, detection, analysis and recovery

Computer technology has changed our life fundamentally and computer systems have become indispensable for the proper functionality of our society. On the other hand, the rapid increase of the society's dependence on computer systems brings great interest to break in and attack those systems. Computer software determines the functionality of computer systems and is the primary target of attacks. With today's pervasive presence of computer systems and network connections, securing critical software from attacks has become an extremely important problem and has never been as challenging.; Critical software faces both software-based attacks and hardware-based attacks. Software-based attacks may come from other malicious software. For example, attacking software may read/write victim software's address space through flaws in process isolation. Software-based attacks may also break software by exploiting all kinds of vulnerabilities in the victim software, such as notorious buffer overflow vulnerabilities. Hardware-based attacks break software by utilizing specialized hardware and attacking the system on which the software is running, such as snooping system buses during the execution of the software. Software attacks are more commonly known. However, combating hardware attacks has been an extremely important problem in secure embedded systems domain, such as smart cards, and is becoming more and more relevant in general purpose computing domains.; In this dissertation, we propose RADAR---compileR and micro-Architecture supported intrusion prevention, Detection, Analysis and Recovery. RADAR is an infrastructure to help prevent, detect and even recover from attacks to critical software. Instead of being a purely software-based approach or a purely hardware-based approach as in previous approaches, our approach emphasizes collaborations between compiler and micro-architecture to avoid the problems of purely software or hardware based approaches. Our infrastructure is based on micro-architecture level support and has its security rooted in hardware. At the same time, we call for compiler assist whenever it is necessary, such as to obtain expected software behavior, or whenever it is helpful to reduce the complexity of the micro-architecture support. With both micro-architecture and compiler support, our infrastructure can defend against both software and hardware attacks with superb security strength but reasonable hardware and performance cost.; We believe that a purely software-based approach is not able to meet the security challenges faced by critical software. First, a purely software-based approach can be easily reverse-engineered and then cracked. In addition, a purely software-based approach cannot defend against hardware attacks thus is not applicable in situations where hardware attacks are real threats. More importantly, security operations implemented in software are much more expensive than the hardware implemented version. The potential performance penalty greatly limits the security strength that a software-based approach can achieve. Overall, a purely software-based approach may not be able to achieve a satisfying security guarantee for critical software.; Thus, we believe that it is time to call for micro-architecture level support for software security. With hardware support for cryptographic operations, such as encryption, decryption and hashing, our infrastructure can achieve strong process isolation to prevent attacks from other processes and to prevent certain types of hardware attacks, such as using specialized hardware to read/tamper the system data bus traffic or the external memory system directly to evade process isolation mechanism completely. Moreover, we show that an unprotected system address bus leaks critical control flow information of the protected software but has never been carefully addressed previously. The information leakage could facilitate an attack and bring significant damage to both code an...