Research On Code Obfuscation Based On Virtual Machine-based Code Protection

Reverse analysis and protection of software are in the game forever.On the one hand,the traditional code obfuscation techniques no longer resist the powerful reverse analysis software.On the other hand,the virtual machine(VM)-based software protection technology makes the existing dynamic and static reverse method failed.However,with the development of the VM-based protection code reverse analysis technology,software using a single basic VM-based protection model can already be reversed.Focusing on the key links and techniques of reverse analysis of virtual machines,such as instruction reduction techniques,byte code disassembly techniques,and semantic analysis techniques,we designed two effective obfuscation algorithms to address the issues of code obfuscation and VM-based software protection.Besides,an enhanced prototype system was designed and implemented.In this paper,the main work includes:1.To withstand the instructions reduction,we proposed a novel approach of instructions reordering based on instructions swapping.Firstly,the formal definition,proposed by Wroblewski G,is improved to solve the sufficient condition of instructions swapping.On the basis of sufficient condition,we proposed a novel approach of instructions reordering based on Simulated Annealing(SA for short).The handlers used as the object,experiment results show that instructionsreordering is effective and applicable for anti-reversing.At the same time,the algorithm can be applied in the traditional code obfuscation technology,which can resist auto-reverse tool analysis and artificial reverse analysis to a certain extent without paying the time and size cost,and increase the static diversity of the sample.2.Aiming at combating the virtual instruction analysis and semantic analysis,an improved scheme of virtual registers rotation was designed.Concerning about the virtual registers,we transfer the mapping relation between the virtual registers and the op-code of the bytecode during executing,which means the uncertainty and complexity of the data flow during interpretive execution of the bytecode.In addition,we propose three policies to address the problem that how to choose the length of rotation for each bytecode,which grows complexity of the protection.3.Design and implement a software protection X86 VM prototype system.Overall system design and implementation and the aim goal of the system are given.We carry on the detailed elaboration of each function module based on the ideas of hierarchical and modularization,and complete on its validity through the experiment verification and analysis of the performance.The experimental results show that the protection for software implementation X86 VM system is effective,reducing the efficiency of protection for the VM code reverse analysis.Finally,a conclusion and a discussion of the future research are given.