| Zero day attacks and hidden Malware pose a grave threat to computer users. To date, widespread security measures such as anti-virus packages and firewalls have proven to be ineffective in guarding against these types of malware. New security measures are essential to secure computer systems, protect digital information, and restore user confidence.;Security mechanisms which are able to differentiate regular (normal) behavior from malicious (abnormal) behavior promise new ways to effectively detect, counter and ultimately prevent the execution of zero day attacks and hidden malware. In this thesis we explore the utility of different software semantics for detecting malicious behavior. Our methods significantly improve upon previous work done in the area of Host-based Intrusion Detection Systems (HIDS). We present novel methods which utilize semantics available in different abstraction levels to detect malicious behavior and provide distinct advantages when compared to the current state of the art HIDS.;Our first approach, Tracks, is able to differentiate normal from abnormal behavior by extracting high-level semantics (application and operating system semantics) obtained and analyzed during runtime. Tracks is designed to accurately identify and capture Trojan Horses, Backdoors and includes a new security policy engine. We demonstrate the utility of this approach and report on both detection rates and performance impacts.;VGuard, our second security mechanism, utilizes the VMM (Virtual Machine Monitor) layer to extract very low-level semantics during runtime. VGuard can overcome some of the limitations of the semantic gap imposed when working at this level of abstraction by employing advanced data mining techniques. When we combine VMM profiling with sophisticated feature-based machine learning algorithms, we are able to accurately identify security intrusions in compute-server applicances, while introducing minimal execution overhead. |
