Anomaly Detection Methods For Host-Based Intrusion Detection Systems

The dissertation concentrates on two types of intrusion detection techniques that are used to detect suspicious activities at the host level. One is anomaly detection of user behaviors using shell commands as audit data. The other is anomaly detection of program behaviors using system calls as audit data. The followings are the main research work and innovative points in the dissertation.(1) The standard structure and working procedure of intrusion detection systems (IDSs) are introduced. Some existing intrusion detection methods are analyzed and compared.(2) A new anomaly detection method based on machine learning is presented, and an anomaly detection system based on the method is designed. In the method, multiple dictionaries of shell command sequences of different lengths are constructed to represent the normal behavior profile of a valid user. During the detection stage, the similarities between the command sequences generated by the monitored user and the sequence dictionaries are calculated. The similarities are then processed and act as the measure to judge whether the behaviors of the monitored user are normal or not.(3) An approach to anomaly detection based on hidden Markov models (HMMs) is proposed. The approach constructs a specific HMM to represent the normal behavior profile of a valid user. Because the collections of observations corresponding to different states are mutually disjoint, the parameters of the HMM can be estimated by a sequence matching algorithm which is much simpler than the classical Baum-Welch algorithm. A decision rule based on the probabilities of short state sequences is adopted.(4) The application of Markov-chain models to anomaly detection of user behaviors is studies, and an anomaly detection method based on Markov-chain models is derived. The method takes the frequency distributions of the observed events into account, and is especially effective when sufficient training data is available.(5) Two methods for anomaly detection of program behaviors are presented, and an anomaly detection system based on the two methods is designed. The methods are both based on data mining (DM) technique, and use sequence patterns to represent the normal behavior profile of a program according to the supports or confidences of the patterns in the training data. In the detection stage, the behaviors of the monitored program are classified by pattern matching.(6) Two approaches to anomaly detection of program behaviors are presented which both use a Markov process to represent the normal behavior profile of a program. The performances of the approaches are respectively tested by experiments. The results show they can both provide high detection accuracy.