Research On Instruction-level Tracing And Replaying Techniques Of Executable Files In Windows

Software security is a core component of computer system security. As an important technical means, software reverse analysis technology can be used in software performance test, vulerability detection, etc. So, it has important significance to do thorough research on dynamic trace debugging technique, which is a key technology in software reverse analysis.In this thesis, the concept and interrelated application aspects of software reverse analysis were firstly introduced. Based of the shortcomings of traditional software debugging methods, we put forward the instruction-level tracing and replaying techniques. At first, Windows debugging interface and dynamic binary analysis technologies were compared, and the latter was chosen as the basic technology of our research. Subsequently, we designed a simulation engine to load the object program and simulate the binary code. Thus, object program's execution details can be monitored at runtime, and a complete trace of a program's user-mode execution can be collected. Due to the complexities of memory-accessing instructions, we presented a real-time memory-accessing information recording technology on the basis of dynamic binary instrumentation. For keeping the tracing overhead for both space and time low, the mechanism of Event Trace for Windows was introduced to our work, and a method based on ETW was proposed to efficiently store and retrieve the program trace. Furthermore, considering the practical needs of software reverse analysis, ollydbg's extension and simulating technology were combined together to propose a flexible approach with threads replacing technique to record particular program slice's trace. After tracing, we did many researches on replaying execution trace deterministically in instruction level, and dynamic process can be analyzed offline. In addition, a trategy was presented to re-execute program trace dynamically and automatically.Based on the researched techniques, we designed and implemented an assistant debugging tool named as TrackReplay, which is composed of trace recording and replaying modules. The performance of this tool was tested with many actual examples, and the results showed that our tool has reasonable runtime overhead in space and time. Moreover, to satisfy various debugging needs, our tool provides a fine interface for extension.Finally, a buffer overflow vulnerability(MS08-067) reported by Microsoft was analyzed in detail using TrackReplay, the result demonstrated that the instruction-level tracing and replaying techniques are very useful in improving ability and efficiency of software reverse analysis.