Research On O-LLVM Deobfuscation Solution Based On IDA Microcode

With the rapid development of computer software and network technology,malware has become an important factor affecting network security.To make the analysis more difficult for network security researchers,malware developers not only use traditional layout obfuscation,data obfuscation,control flow obfuscation,and obfuscation prevention techniques,but are also more enthusiastic about O-LLVM(Obfuscator-LLVM)customized code obfuscators with better protection effects.Aiming at the limitations of previous O-LLVM deobfuscation solutions in customized obfuscators,we conducts in-depth research on technologies such as code obfuscation,intermediate language,simulated execution and taint analysis.Then we solves some problems of previous deobfuscation solutions,such as single instruction set architecture,inapplicability of customized obfuscators,no support for path branch obfuscation,and lack of dynamic capabilities caused by intermediate languages,which significantly improves the portability,compatibility and applicability of O-LLVM deobfuscation solution.The specific work is as follows.A real block identification algorithm and a multiplexed block segmentation algorithm compatible with customized obfuscators are proposed,and then combined with the intermediate language IDA microcode,a control flow deobfuscation solution,Bin Deob,is designed and implemented.This solution is suitable for standard obfuscators and customized obfuscators,and can deobfuscate malware running on various common instruction set architectures such as ARM32,ARM64,X86 and X64.Experiments on C/C++ classic confusion benchmarks and public high-risk security vulnerability dataset show that the average similarity of the control flow graph between the Bin Deob deobfuscation program and the original unobfuscated program is 98.9%,which is better than the deobfuscation solutions such as Di ANa and SATURN in the references.In addition,pseudo-code similarity is introduced as an evaluation index,and the average pseudo-code similarity between the pseudo-code obtained by Bin Deob’s deobfuscation and the program source code is 97.6%.A control flow reconstruction algorithm suitable for branch target obfuscation is proposed.The first simulation execution engine of IDA microcode is implemented,and we call it Bin VM.Combined with static taint analysis technology and IDA microcode,a branch target deobfuscation solution,Br Deob,is designed and implemented.This solution can deobfuscate the malware whose path branches were obfuscated.In other words,it can obtain the obfuscated branch target address,reconstruct the control flow,and finally generate the deobfuscated pseudo-code.In addition,the simulation execution engine Bin VM endows Br Deob with dynamic capabilities.As a result,Br Deob can also solve some code obfuscation problems such as opaque predicates and string encryption without running malware.Experimental results on public datasets show that the average values of the control flow graph similarity and the pseudo-code similarity between the Br Deob deobfuscation function and the original function are 98.5% and 96.9%,respectively,which demonstrates the effectiveness of Br Deob against real threats.