Research On Key Techniques Of Efficient Protocol Software Security Analysis

The protocol software is widespread and ubiquitous in network applications with the rapid development of Internet.At the same time,there are more and more attacks against the protocol software,which mostly are remote attacks exploiting the vulnerabilities of protocol software.The consequences are extremely serious.However,it has always been a very difficult problem to analyze the security of protocol software.Protocol software manufacturers often utilize a variety of techniques to analyze the security to prevent the protocol software from being attacked,hoping to find and eliminate as many vulnerabilities as possible.Meanwhile,the hackers may also try to discover the security vulnerabilities and make use of them to conduct attacks.Therefore,it is of great significance to conduct the research on efficient security analysis of protocol software for timely detecting the vulnerabilities and preventing the hacker's attacks.This thesis investigate key techniques for efficient protocol software security analysis.It is well known that current security analysis techniques for protocol software suffers from many problems,such as low universality,low test depth,low test efficiency and low degree of automation,etc.We also lack automated methods for efficient security analysis of protocol software.The conventional symbolic execution method is widely utilized for the security analysis of protocol software,however,it can only test the single packet input with shallow test depth.The conventional fuzz testing method is also applied to the security analysis of protocol software,but it is characterized with the strong system randomness and low code coverage.Given these problems in the security analysis of protocol software,this thesis studies the depth test method for multiple-input state protocol software and also the efficient test methods for protocol software.The goal is to improve the universality and the automation degree of security analysis of protocol software.The main work and novelty of this paper include the following aspects:A symbolic execution method based on multiple-packet selection is proposed.This method combines the selective symbol execution method,which enables the symbolic execution method to deal with the multiple packet inputs of the protocol software.Firstly,the input data packet is processed symbolically,and the system calls related to network applications are intercepted.After that,the symbolic input data is forwarded to the testing target.Then,an improvement is made based on the data-packet selection symbol execution method to make it deal with the inputs of multiple packets;A multiple data-packet selective symbolic execution method is proposed,which improves the test depth of the security analysis of protocol software.A model-based directed symbol execution method is proposed.This method utilizes the interaction characteristics of the protocol software and the known protocol specification information.Firstly,an efficient state machine learning algorithm for protocol software is proposed to deduce the protocl state machine model.Then,the protocol model is utilized to guide symbolic execution of the protocol software,so that the symbolic execution engine can quickly reach the protocol state that needs to be tested.The new method alleviates the problem of state-space explosion of symbolic execution,and improves the test efficiency of the security analysis of protocol software.A path reduction method based on joint symbolic execution is proposed.This method reduces the path-space of symbolic execution to alleviate the path space explosion problem and improves the test efficiency of security analysis of protocol software by synchronizing symbol executions of sender and receiver of protocol software as well as taking into account the constraints between the sender and receiver,so that the security issues of protocol software in interoperability can be found.A fuzz testing method for protocol software based on modular protocol specification language and genetic algorithm is proposed.The modular protocol description language and the genetic algorithm model are combined to improve the code coverage of fuzz testing and improve the test efficiency of the security analysis of protocol software.As a complement to symbolic execution method,this new method can be utilized in the case where the constraints of symbolic execution cannot be solved.