Research Of Reverse Analysis On Self-Modifying Code

Malware(Malicious Code or Malicious Software) creeps into users' computers, collecting users' private information,wrecking havoc on the Interact,has become the centerpiece of most security threats on the Internet.With the popularity of computer and the development of the Internet,the damage caused by malware is also more and more serious.To enhance the emergency response speed of network attacks that malware actualized,we must analyze malware rapidly and effectively.Malware analysis is an essential technology that extracts the runtime behavior of malware,and supplies signatures to detection systems and provides evidence for recovery and cleanup.To hinder malware analysis and make the analysis more difficult,malware writers usually have their programs heavy-armored with various anti-reverse engineering techniques.Such techniques include code encryption,metamorphism and binary code packing.Unfortunately,existing techniques for detecting malware and analyzing unknown code samples are insufficient and have significant shortcomings. Existing solutions are either unable to handle novel malware samples,or vulnerable to various evasion techniques.To meet the needs of malware analysis and security evaluation,in this paper,we analyze the key anti-reverse engineering techniques and related work of malware analysis,and the implementation mechanism of typical Self-Modifying Code(SMC) thoroughly.Then based on above analysis works and results,we propose a reverse engineering approach for typical SMC based on emulator,which is motivated by the intuition how to combine static analysis and dynamic analysis effectively.Mainly has done the following several aspect works:First,the implementation mechanism of typical SMC is analyzed thoroughly,and a primary model of SMC is proposed.We model and classify typical SMC according to generation mode,modification mode and storage mode of dynamically generated code.The research and its application of the mechanism provide theoretical foundation and guideline for the study of the reverse engineering techniques against typical SMC.Second,a fully dynamic approach for extracting the original hidden code (dynamically generated code) and additional information useful for further analysis of packed executable binaries is presented.In this paper,we present a binary extraction technique which is fully dynamic and thus does not depend on the program disassembly or the known signatures of packing techniques.We also show that our proposed technique can extract the original hidden code and data.In addition to extracting the hidden code,our proposed method can provide additional information on the packed executable binaries.It can identify the exact regions of memory where the hidden code and data reside.By tracking the newly-written memory areas of the program,we can distinguish newly-generated code and data at run-time from the packed executable binary,and thus obtain the exact regions of them.Third,a fully dynamic approach for identifying and extracting dynamically generated code and additional information useful for further analysis of packed DLLs (dynamic linker libraries) is presented.In this paper,we propose a technique to extract the hidden code by loading a DLL and triggering and monitoring the execution of the entry-point function and exported functions of packed DLLs.By monitoring all memory operations and control transfer instructions,our approach extracts the original hidden code which is written into the memory at run-time.Fourth,a technique for reconstructing a SMC binary for static analysis is proposed.Our proposed technique constructs a binary based on the original SMC binary,the hidden codes extracted and the records of control transfers,by patching the hidden code extracted on the packed binary and restores the control transfers to generate a binary for static analysis.Our proposed technique modifies the original binary to generate equivalent static code without altering its origin program behavior. The reconstructed binaries can be successfully analyzed by static analysis tools,such as IDA Pro.Fifth,a system by exploring multiple execution paths for malware analysis based on code coverage is proposed.Our proposed method reduces the times of some paths explored and improves the analysis efficiency and increases the coverage of malware by way of labeling control flow decision points(branching points).Sixth,an automated framework for extracting hidden code and reconstructing SMC binaries is designed and implemented.Applying our above proposed technique, we build a framework for automatically examining SMC binaries,extracting their original hidden code and reconstructing a binary based on the extracted code and additional information.Based on the prototype,we have successfully done a series of experiments on the analysis of typical SMC binary.We also present the evaluation results of the framework,demonstrating that it is applicable to analyze typical SMC binary.