Dynamic Software Analysis Based On Virtual Machine

Dynamic software analysis approach is an important way to understand software behavior and infer internal algorithm, which is necessary mean to detect the malicious code of software. Dynamic analysis based on virtual machine technology can be applied into dynamic binary instruction flow research and dynamic byte instruction analysis. This thesis proposes two kinds of innovative dynamic software analysis approaches based on virtual machine, which are dynamic binary code differential analysis and Python interpreter security enhancement, to evaluate the security and potential risks of binary software and Python byte code software.We propose a dynamic binary flow differential analysis approach, design and implement the prototype system, which is made by modifying the Bochs virtual machine. Firstly, we define the binary differential analysis approach that infers the difference between the different dynamic binary flows as various input data. Our approach could effectively extract the sensitive information from malicious code and make the function module or data spread understood. Finally, we provide an experiment based on differential binary analysis system, which validate the capability and efficiency of the approach.According to Python interpreter working mechanism and virtual machine technology, we design and implement the PyXhon security evaluation system to detect the potential security risks in Python software. We propose the Function Oriented Analysis, which developers use to monitor all function-call procedures; dynamic Byte Instruction Trace Analysis, which infers the behaviors of importing modules and accessing private DLL; and security policies, which provides strategies to accept or reject extensions. These security mechanisms do not require Python language features so as to be completely transparent to Python applications.The achievements of this thesis are following:(1) Design and implement the dynamic binary instruction flow analysis system, that captures the dynamic instructions and infer the software behaviors. This evaluation system is implemented with Bochs emulation, which is transparent to upper-level binary software, effective and precise.(2) Propose the dynamic binary differential analysis approach. Define the dynamic binary instruction differential model; analyze the efficiency and accuracy of the prototype system; evaluate the capability to tackle the obfuscation and encryption technology in binary code.(3) Discuss and analyze the application and mechanism of Python programming language and third-party extensions; propose the innovative function-oriented model and Python byte code inference model; describe the detail and experiment of PyXhon security evaluation system.