Semantic approaches for intrusion detection and prevention

This dissertation presents research on semantic approaches to behavior analysis. It aims at enhancing computer defenses making them invulnerable to new, mutating and obfuscated malware. Both signature and anomaly based approaches are proposed and used to develop scalable IDSs. The presented approaches raise detection semantics from behavior to functionality allowing for identifying some classes of malware achieving the same specific malicious goals. Moreover, an approach for modeling self-replication attacks and automatic immune response to these attacks was studied and verified on simulation models.;A comprehensive taxonomy of malicious functionalities in typical malware is presented. It allows for understanding and classifying the essence of maliciousness that serves as the detection criteria in the proposed signature based approach.;Anomaly based approach for detecting self-propagating malware in the behavioral domain is introduced. The approach utilizes non-stationary Markov models for normalcy profiling and leverages the anomaly propagation concept to mitigate the false positive problem. The research results indicate that application of this approach can provide a high level of confidence in attack detection. The experiments showed extremely low false positives, anomaly propagation was detected at early stages of the worm attack, showing high dependability and low detection inertia of the IDS.;The signature based approach allows for detecting malicious functionalities in the system call domain using rather generic and highly semantic signatures. Such an approach is superior to existing behavior based techniques in addressing behavioral obfuscations and multiple functionality realizations. The proposed IDS detects intrusions at the semantically highest level---functionalities, which semantically is higher than behavior. Indeed, behavior is merely a manifestation of one of the realizations of functionality that in its turn might be easily obfuscated. The proposed approach addresses three interrelated aspects: signature expressiveness, behavioral obfuscation and run-time signature matching efficiency.;Finally, the proposed techniques have been implemented in a prototype IDS and evaluated on dozens of malware and hundreds of legitimate programs. The experiments results indicate low rate of false positives and negatives, and low execution overhead. Such results suggest that detecting malicious functionality presents a sufficiently dependable and efficient method for distinguishing malware from benign software.