Intrusion detection/prevention using behavior specifications

In this dissertation, we present an approach to detect attacks on computing infrastructures and launch responses that prevent or minimize the damage caused by these attacks. Our approach is specification based, in which, security-relevant behavior of a system is specified using a high level specification language. Attacks are detected as deviations from the specified behavior by an enforcement algorithm. Previous work on specifying and enforcing security behavior fall short in addressing the key requirements of such an approach which are: an expressive/concise language with unambiguous semantics and an efficient enforcement algorithm which is correct with respect to the semantics of the language. The result is that in these approaches expressive behavioral properties cannot be specified, enforcement of them is not efficient, or they do not provide sufficient confidence in their enforcement algorithm to launch responses to prevent attacks.; regular expressions over events (REEs) and an enforcement algorithm based on the computational model of REEs, called extended finite automata (EFA). We present the unambiguous/precise semantics of REEs and prove the correctness and completeness of the enforcement algorithm with respect to them. This provides assurance that specified behavior is the enforced behavior. We also developed an algorithm that translates an REE specification into a fast pattern matching automaton that forms the basis of the efficient enforcement algorithm.; In addition, in this dissertation, REEs and EFAs were used to develop a prototype intrusion detection/prevention system for the UNIX operating system. The system was designed based on the following two observations: regardless of the nature of attack, damage will ultimately be caused by the system calls made by the attacked processes, and no damage can be caused if the program is behaving normally. We developed a concrete language based on REEs called behavior modeling specification language (BMSL) to specify program behavior as well as responses if any attacks are detected. BMSL specifications capture behaviors of programs on the UNIX operating system as sequences of system calls and their arguments, made by these programs. The enforcement mechanism is based on system call interposing. Intercepted system calls are redirected to the enforcement algorithm corresponding to the process making the system calls. We describe the problems with the two interposition approaches most commonly used in intrusion detection techniques, kernel and user-level, and propose a novel approach called hybrid interposition. We address the key challenge of this approach which is splitting the functionality of the enforcement algorithm between the kernel and user-levels. (Abstract shortened by UMI.)...