Research On Reverse Analysis Techniques Of Hidden Code In Executables

Software security is a core component of information security, as an important technology means, software reverse analysis can ensure software security, has significant application value in software vulnerability discovery, malware detection, etc. But as the development of anti-reversing technology, more and more software use shell, encryption, obfuscation and other technologies to hide their code, using only static or dynamic technology could hardly distill and analyze hidden code among programs. Using static and dynamic together is a trend in software analysis, so do thorough research on this technology has great meaning.The common methods in software reverse analysis and existing anti-reversing analysis technologies are firstly summarized, following which the problems of existing code analysis technologies in handling hidden code are compared, and then the hidden code dynamic capture approach based on static control flow analysis is proposed. To obtain the precise control structure for dynamic instrumentation, a control flow traversal approach based on prioritizing accurate code identification is designed through analyzing the limitations of common static control flow parsing in handling the obfuscated code; Aiming at the problem of modifying return address to hide code, an approach of return address analysis based on the reduced backward slice is proposed to solve the determinant of return address in function calls;On the basis of static analysis, a dynamic binary instrument approach is addressed based on program structure, and dynamic capture of hidden code and pre-execute analysis is implemented; In order to overcome the restriction of anti-debugging to dynamic analysis, the thesis also do research on anti-anti-debugging technologies which based on the detections of debugging flags, system API and breakpoint.The thesis designes and implements a prototype system of reversing binaries which can find hidden code in executables, then tests the function and performance, the testing results show that the system can get hidden code in executable program effectively, and has great application value in software which has applied anti-reversing technology.