Reverse Extracting Of Protocol Model Based On Dynamic Binary Analysis And Its Application

With the rapid development of Internet, network based applications have increasingly gone into every field of the reality society. Network protocols, especially cryptographic protocol, as the fundament of computer network are so important in their usability, reliability and security. For this reason, researches related to automatic protocol reverse engineering are becoming hotspot and major direction. As an important objective of protocol reverse engineering, protocol model abstractly describes the dynamic network behaviors of application, and has important application value in the fields of protocol security analysis, program verification, analysis of network behavior, vulnerability discovery, protocol fingerprinting.This paper takes how to extracting protocol model as basic goal, aims at the difficulties in deducing message fields and semantics, parsing encrypted steam, inferring protocol sequential logic and state transition, analyzing sophisticated binary code of network application, proposes a method for extracting protocol model based on dynamic binary analysis technique, which mainly concerns about how to reverse derive protocol format, protocol model, protocol specification. And on this basis this paper studies a method under the guidance of protocol model for mining protocol deviation, and proposes a method for extracting and identifying program protocol fingerprint based on protocol deviation. The main contributions and innovations of the thesis is:(1) In-depth and extensive summarization of the related works in protocol reverse-engineering and dynamic binary analysis. For the problems exist in protocol verification, program network behavior analysis and protocol vulnerability discovery, the thesis first introduces the technology used in protocol reverse-engineering from views of network trace and host, then classifies the existing solutions and summarize their advantages, disadvantages, and application range, consequently explains the study works. Aiming at the basic technical support, some theories related to dynamic binary analysis are introduced, such as taint propagation and Dynamic Binary Instrument(DBI). Meanwhile, the advantages and disadvantages of kinds of dynamic binary analysis tools and platform are summarized.(2) A framework oriented to reverse parsing of message format based on dynamic binary analysis has been proposed. Analyzing encrypted stream, message field and the corresponding semantics have been the challenges of protocol reverse engineering. The fundamental cause is reverse analysis method itself and difficulties in obtaining protocol information. Combination of semantics in program’s encryption and decryption behaviors, a function level and instruction level based method for reverse reasoning message field has been proposed, overcoming the problem of low precision and short application range of function level tainted analysis method, and the difficulty of reverse deriving semantics in instruction level tainted analysis method. Based on that, a method for reverse parsing cryptographic protocol were proposed, which partly overcome the difficulty of network based method in analyzing encrypted stream.(3) A distributed protocol model reverse-inference technology based on message interaction graph describing network behavior has been proposed. Protocol model abstractly describes dynamic network behaviors of network applications, however, modern network protocol, especially security protocol commonly have complicated temporal logic and state transition. Inspired from theory and method in state machine, this paper proposes a message interaction graph based approach to extract protocol model from multi-role protocol applications, based on this, a conversion algorithm has been proposed to automatically describe the derived protocol model as formal specification.(4) In the application fields of protocol model, a protocol model guided automatic approach to mine protocol deviation has been proposed. Protocol deviations describe network behaviors discrepancies between each version of protocol implementation application. In consideration of protocol deviations in program verification and protocol fingerprinting, this paper proposes a method to discover deviations existing in protocol application program by conducting a serious of iterative test, and the derived model will be corrected along with the iteration is performed.(5) A new program fingerprint extraction and recognition approach based on protocol deviations has been proposed. Due to traditional extraction of protocol fingerprint spends lots of time and manpower, combining the feature of protocol deviation, this paper first proposes a new approach for extracting and recognizing program fingerprint. The key idea here is extraction of protocol fingerprint by observing the dynamic progress of message processing, therefore, this approach can be used in identifying fingerprint of cryptographic protocol applications. According to deviation session stream and deviation message, this paper first combines TPFSM of session stream and the features of protocol deviation responding message, proposes a method for fingerprint extraction from protocol. In the aspect of protocol recognition, this paper first introduces the concept of session steams coding and SHINGLE, then proposes a SHINGLE based algorithm for feature matching of session stream, and a regular expression based method for message matching.The research of the thesis is effective practice and exploration in the field of protocol reverse-engineering. Research achievements are significant in some application field such as program verification and network behavior analysis,and will actively boosts the development of network security technology.