Software and hardware techniques for masking security vulnerabilities

Security problems in computer systems are caused by a variety of design deficiencies, implementation defects and, as this thesis shows, by transient errors. The thesis first studies real world security vulnerabilities via an analysis of available security data and experimentation. Results of the study show that: (i) Nearly 60% of the successful attacks exploit design defects in target applications that allow attackers to tamper with critical control information to seize the application execution; and (ii) Naturally occurring transient errors can infrequently corrupt security sensitive code in such a way as to leave the software in a totally vulnerable state open to attacks. A stochastic model in conjunction with real data demonstrates that, although infrequent, such problems have a finite probability. The thesis then proposes two algorithms—Transparent Runtime Randomization (TRR) and Control Data Randomization (CDR)—to protect an application against security attacks. The two algorithms are implemented by instrumenting the operating system, runtime system and compiler, and are experimentally evaluated against a wide range of security attacks. The thesis also explores the potential of architectural support for security. Three hardware security modules—Secure Return Address Stack (SRAS), Memory Layout Randomization and Data Dependency Tracker—are implemented and evaluated in a processor-level framework called Reliability and Security Engine.