Research On Virtualized Code Deobfuscation

Sensitive code nowadays is often protected against reverse engineering with code virtualization techniques,which convert original opcodes(e.g.,x86/x64 instructions)into virtual bytecodes that are only interpreted by a proprietary virtual machine.Those virtual bytecodes and the virtual machine itself are also unique for every protected application,avoiding a general attack.While code virtualization protects commercial software products,it also helps evade malicious code detection and makes obfuscated code hard to maintain or debug.Moreover,current virtualization schemes generally introduce significant extra runtime overhead.Therefore,it is expected to develop a generic binary code analysis approach against code virtualization.In this paper,we conduct an in-depth study on code virtualization protectors and propose a generic de-obfuscation approach that aims to simplify those virtualized code.Our approach first pinpoints the interpretation procedure of the embedded virtual machine and partitions handlers of the embedded VM,and then employs a VM-state based handler translating,which represents the VM-state-updated behaviors of handlers using intermediate representation.Finally,the translated operations of each handler is optimized and the execution of heavily obfuscated embedded bytecode is rewritten as a simpler one.We build Nightingale,a binary translation tool,to fulfil this de-obfuscation process automatically with x86 and x64 binary executables.We test our approach on three commercial code obfuscators and a set of home brewed code virtualization schemes.The results demonstrate that this kind of virtualized code can be simplified effectively.