Research On Malware Analysis Based On Structure And Behavior

With the Internet becoming an indispensable part in people's daily life, the Internet significantly benefits people's life, but it also makes people frequently attacked by malware. When the generating and propagating of malware becomes a black industrial chain, the amount of malware becomes greater and its producing speed becomes more rapid. Therefore, the malware threatens the Internet security of our country.With the development of Internet technology and the appearance of the black industrial chain, the self-protecting of malware is getting matured. This makes the current malware analysis technology not satisfy the need of malware analysis and detection. Therefore, there are great challenges for current malware analysis technology. The work of this thesis can be divided into the following four parts.1) Malware detection based on binary file structureWe present an accurate framework that automatically extracts distinguishing features from portable executables (PE) to detect unknown malware. The distinguishing features are extracted using the structural information standardized by the Microsoft Windows operating system for executables, DLLs and object files. We follow a threefold research methodology: (a) identify a set of structural features for PE files which is computable in realtime; use an efficient preprocessor for removing redundancy in the features'set; and select an efficient data mining algorithm for final classification between benign and malicious executables. We have evaluated our framework on two malware collections, VX Heavens and Malfease datasets. The results of our experiments show that our framework achieves more than 99% detection rate with less than 0.5% false alarm rate for distinguishing between benign and malicious executables.2) Browser malware detection based on memory structureBrowser attack has become the primary infection vector of malware, and these malware could be used to build botnets, sending spam email, host phishing host, or launch a DDoS attack. Heap spraying is the most used browser attack type in the wild, which can improve the attack reliability to bypass the protection of Address Space Layout Randomization. A typical heap spraying attack allocates a large number of string objects that containing NOP sled on the heap.In this paper, we describe HSDSD, a runtime monitoring infrastructure that detects heap spraying attack in browsers. HSDSD is implemented and integrated in Firefox browser. It monitors JavaScript string allocation in Firefox to detect NOP sled through Sampling Sled Distance Measurement. The evaluation result shows that the detection rate of HSDSD is 99.7% while the performance overhead is 3.5% on average.3) Malware unpacking method based on system call behaviorSoftware packing is an obfuscation technique to protect software against reverse engineering, but it is commonly used to hide malicious code to anti-detect as well. Environment-sensitive packing techniques are able to check whether the run-time environment is suspicious, then malware can dynamically change the unpacking behaviors according to the environment. While many unpacking tools were proposed, such as static unpackers and dynamic unpackers, the existing solutions are either unable to handle unknown packing techniques, or vulnerable to various environment-sensitive techniques. In this paper, we propose a new unpacking approach based on environment-sensitive analysis. Our approach analyze the difference of the system call information of the packed malware to unpack it, and we present a prototype system, called DscUnpack and apply it to packed samples. The experimental results show that DscUnpack can effectively identify and unpack samples.4) Malware analysis method based on objects operationModeling accurate software behavior based on system call has become an open issue. The traditional malware behavior models may be evaded by the attacker who uses the system call obfuscation technique. Therefore, this paper presents a malware behavior model based on system object operation. The experimental results show that our approach can effectively detect malware.