Host Anomaly Detection System Based On Program Behavior

Because anomaly detection can find new intrusion, it has become a hot topic in intrusion detection. At present, most of the attacks exploit the vulnerabilities or flaws of the privileged processes in computer. Compared to user behavior profiles, program profiles are more stable over time because the range of program behavior is more limited. Furthermore, it would be more difficult for attackers to perform intrusive activities without revealing their tracks in the execution logs. Therefore program profiles provide concise and stable tracks for intrusion detection. A program's normal behavior is characterized by its local ordering of system calls, and deviations from their local patterns are regarded as violations of an execution program. So, it's a good idea to find intrusion by watching program profiles. But it still cannot meet the need of application, for its efficiency is low. This thesis builds a host anomaly detection system based on program behavior. The system can efficiently find intrusion by analyzing the program behavior and can be applied to detection on line.In order to analyze the present research situation in this field and identify the key points of our research and efforts that we should make in the future, first of all, this thesis not only summarizes the research progress of anomaly detection based on program behavior in two aspects: the angle of looking at normal and the methods of making models, but also analyzes error characteristics and complexity of main methods.The first condition for establishing an exact program's normal behavior model is correct and sufficient training data, which can affect the accuracy of anomaly detection system. Therefore, this thesis makes a study of the methods of collecting training data. Analyzing the error characteristics of available methods, we collect the data in the real environment, for only these program behaviors are the normal ones. In order to include all the normal behaviors, this thesis studies the conditions for the end of collecting training data. Meanwhile, because the real environment may be intruded, we delete the anomaly behaviors. Methods are as follows: we establish a normal behavior model called Callgraph with the static method to analyze the program codes. Then we drive Callgraph model using the original training data. Once some sequence cannot be accepted, it will be deleted. The experiment results show that most of the intrusion sequences can be deleted with this method.Anomaly detection model is the kernel of the whole anomaly detection system. In order to improve the accuracy and maintain an acceptable algorithm complexity, we propose a new method for anomaly detection based on Character Patterns and Markov Chain Model after analyzing the available anomaly detection models. In this model, the authors extract the short sequence of system calls as a character pattern if this sequence satisfies the certain support degree, and propose an improved Markov Model CPMC on this basis. When detecting intrusions, firstly, we would use the program trace to match character patterns. Then the authors would calculate the trace's probability under CPMC model. Small probability means anomaly. The experiment results show that the authors can get higher detection accuracy than that with other current single methods. Compared with DBCPIDS, our method has the approximate accuracy but lower computational complexity.At last, we build a host anomaly detection system based on program behavior to test the feasibility of the methods. This part focuses on how the two models, event generators model and Event analyzers model, are formed. Meanwhile, we test this anomaly detection system.