Research Into Malware Behavior Analysis And Obfuscation Detection Technology

A bottleneck in malware detection is malware behavior analysis. It means analyzing the function and attribute on the unkown mal-sample. Static analysis can achieve the integrity in capturing the program information. However, it has encountered some challenge such as packing, encryption and malware obfuscation technology. Dynamic analysis can bypass some problem cased by them, and achieve higher system abstractness. But it has also faced some problem cased by anti-debugging and anti-tracing technology. Some obfuscation technology has been used by malware author to evade signature based detection, such as encryption, polymorphic and metamorphic. In this paper, we addressed on the malware obfuscation detection technology based on the dynamic analysis. Main research contents are as follows:Firstly, some integrity requirements to construct an effective dynamic analysis system were proposed. In order to achieve these requirements, we constructed analysis system based on Intel VT. It divided the target malware and analysis system into different operation system, and traced the runtime system call through virtual machine monitor. In the system, we model the program behavior by maximal patterns, which represents some system function or program module. And we proposed maxmimal pattern mining algorithm to mine the maximal patterns from the system call sequence. A prototype system has been implemented to validate this idea. The result of our experiment has shown that the proposed method can trace the runtime information of program effectively and model the program behavior with high system abstractness.Secondly, three malware similarity calculation methods based on dynamic analysis were proposed. The first is the malware similarity calculation based on the system call sequence alignment, which used evlutioanry similarity algorithm to align system call sequence. The second is the malware similarity calculation based on the maximal patterns sequence alignment, which used maximal patterns mining algorithm to mine the maximal patterns sequence from system call sequence, and used evolutionary similarity algorithm to align maximal patterns sequence. The third is the malware similarity calculation based on the maximal pattern coverage, which model the program by maximal patterns set and compute the similarity score by the insersection between two set.At last, we addressed on three malware obfuscation technology. They are all based on the similarity calculation: the malware obfuscation detection based on system call sequence alignment, the malware obfuscation detection based on maximal pattern sequence alignment and the malware obfuscation detection based on maximal pattern coverage. In order to compare their effectiveness, a same date set was used by them in experiments. The experiment results have shown that the sequence based obfuscation detection method can describe program behavior more accurately and perform better. Especially, the method based on maximal patterns sequence aglinment can reach the lowest false positive rate. We consider it is more efficient.