The Research On Analysis Of Software Behavior And Intrusion Detection Based On Symbolic Execution

Computer science and software has been closely related to people’s life, while itspotential safety hazards have also been attracted more and more attentions. Softwarebehavior analysis technology and anomaly detection can block abnormal behaviorstimely by monitoring attacks from internal and external, to guarantee system securityand reliability.Building normal behavior model of software by using software behavior analysistechnology is the key point for anomaly detection. For the special features of systemcalls, researching anomaly detection based on system call has highly meanings andvalues. Meanwhile, how to obtain anticipated behavior set of software and how tomodel the set are still the main issues. Current modeling methods are mainly focus onsequence model and automaton model. Sequence model lacks of understanding ofprogram structure, so its detection ability is very low. Automaton model also met theproblems of uncertainty, infeasible path and having huge set of states.Modeling methods are mainly focusing on sequence model and automation model.Traditional sequence model met problems in high false alarming rate and unable torepresent program branching and loop structure, so it has been eliminated gradually.Automation can better describe structures of program, enhance accuracy of detection.But automations usually have huge state space, and encounter with infeasible pathproblem.Based on that, we develop an approach of analyzing and modeling system callbehaviors of software, which is based on symbolic execution. First, achieve controlflow graph by static analysis to the program, and build the FSA according to it.Comparing with traditional FSA model based on system call graph, this simplifies thestate space of automations. We also do optimization to the FSA to reduce the statespace again, and increase efficiency of detecting system. Then, we get normalbehavior set by dynamic training. We use dynamic symbolic execution technology toobtain data set for training, and use taint analysis technology to determine variablesneeded to be symbolized. Finally, we use binary instrumentation to do automatictraining. The approach increases the integrity and accuracy of behavior model, anddecreases complexity of building model, as well as eliminates the infeasible path of automation.Based on our research, we implement a prototype of Intrusion Detection Systemcombined with interception and monitoring technology of system call. Test resultsshow that our method can increase the accuracy of detection and decrease the falsealarming rate, without increasing additional overhead to system.