Research On Anomaly Detection Based On Static Analysis

When a program is attacked,it will behave in a manner inconsistent with its binary code, which can be made use of to perform anomaly detection:Firstly,do a static analysis of the binary code to construct a model of the program behavior,and then different kinds of attacks can be detected by monitoring whether the execution of this program deviates from this model. These models usually share the highly desirable feature that they do not produce false alarms but they face the conflict between precision and efficiency:The high precision of the models such as the VPStatic is at the cost of high space and time complexities.In this thesis,we propose a new context-sensitive anomaly detection model based on static binary analysis: STT.This model uses stack walks to eliminate non-determinability and is a provably DPDA which is similar to the VPStatic.STT model replaces the automaton in the VPStatic with a state transition table and all redundant states and corresponding transitions are eliminated,as a result,the model is compacted greatly.According to our analysis,both time and space complexities of the STT are reduced compared with the VPStatic,in particular,in our experiments,the memory overheads of the STT models for the two Linux's self-contained programs(gzip and cat) are less than half of the VPStatics'.STT is also proved that it holds the same precision with the VPStatic.Thereby,the efficiency of the VPStatic is greatly improved without reducing its precision,which alleviates the historical conflict between the efficiency and precision.At the end,a unified formal definition shared by all statically-constructed software anomaly detection models is presented based on Abstract Interpretation Theory.In this definition,program behaviors modeled by different models are considered as approximations of partial trace semantics at different abstraction levels.As examples,abstraction functions corresponding to several classic models including STT are also given in this thesis.In this way,a systematic theoretical framework for the software anomaly detection is established.