Virtual Machine-based General-purpose Automation Shelling

With the further development of information technology, the Intellectual Property Protection of software, as the distillate of knowledge and wisdom, becomes increasingly important. On the other side, malware, emerge in endlessly, who wants to extend the lifecycle of himself, is seeking for the way to product variation to avoid the detection of signature-based anti-virus software.Packer is a program who can keep the target software away from reverse analysis. Aiming attention on the binary code, it is separated with the logic of target with a high stability. So It's popular in Anti-Antivirus field. One malware usually machining hundreds variation by packing himself. According to the report of security manufacturer, the ratio of malware with a packer is up to 95%.In this case, besides increasing the virus library passively, research on unpacking starts at the same time. Nowadays, the method to unpack could only access the specific one. Unfortunately, to get a accurate result, it need much more Manual intervention. A automatic, general unpacker can ease the intension of analyzer and also can be installed on user agent to help AV software.A automatic and generic unpacker based on virtual machine (VAGUnpacker) is presented in this thesis. We have studied the following questions: firstly, we do a comprehensive research to abstract the common characteristic of packer. Then, according to the above knowledge, we present a novel way to solve these obstacles, and show its details. thereinto, we design a light virtual machine to meet the safety and control requirements. Empirical testing indicates that VAGUnpacker can deal with both known and unknown packer independent of packing algorithms and it is faster than existing unpackers. VAGUnpacker can improve the performance and effectiveness of unpacking significantly.This thesis makes the following contributions: 1. Summarize the primary behaviors of packer into the Code Obfuscation, PE Formats Modification and Anti-Technique. 2. A new automatic, generic way to decrypt code in memory on the basis of stack balance role, intersection jump role and the characteristics of entrance. 3. furthermore, for the first time, after locating Import Address Table (IAT) by monitoring all of the call instructions, a forensics tracing technique to restore the items in IAT,which are unmatched with Export Table items of DLL, is presented to obtain a runnable binary.