Malware Detection Based On System Call Dependency Graph

Living in the Internet age, almost everyone’s computer has been attacked bythe malware such as viruses, worms and trojans. People make use of anti-virussoftware to protect against malware. However, a large number of malwarevariants are created by mutation to escape the anti-virus software.Many researchers are focus on extracting features with good generalizationability to detect malware effectively by static methods. But, static methods arevulnerable to code packing or code obfuscation. As confrontation upgrades, staticanalysis becomes more and more difficult. In contrast, dynamic methods are notaffected by these tricks. By running malware in a controlled environment, it cancapture much key information, such as system calls.System call is the interface for applications to interact with the operatingsystem and its execution will usually change the system state. Furthermore, mostsensitive operations of the malware will transferred into system calls, so one canobtain essential characteristics of the running program from system call level.Based on system call sequence, machine learning methods can be used to classifyor cluster a large number of malware, but these methods cannot effectivelyextract the essential behavior characteristics of one malware family. Consideringmany malicious operations are achieved by a series of system calls which havedependency relations, some researchers try to use the system call dependencygraph to characterize malicious behaviors, and to detect or analyze malwareeffectively based on these graphs. The key to this method is the constructing ofthe dependency graph. Moreover, the dependency graph built from the runninglog file contains so much noise information that it cannot be used for malwaredetection directly.In order to solve these two issues, this paper has proposed a new malwaredetection method based on the dependency graph. The method uses dynamic taintanalysis technique to mark system call parameters with taint tags, and then buildsthe dependency graph according to these tags. Meanwhile, in order to extract arepresentative dependency graph, the method uses a graph clustering algorithmto cluster many pruned dependency graphs of the same malware family into aweighted minimal common supergraph, and then detects malicious code by graphmatching. Experiments show that this method has high detection rates andextremely low false alarm rates. Because the construction of dependency graphsneeds to capture system calls and parameters’ taint tags of the running program,this paper has built a dynamic analysis system based on the open-source binary instrumentation platform TEMU to achieve this purpose. Experiments show thatthe system has good tracking results and performance, and it can be used forautomatic analysis of many samples. In addition, the dependency graph cancharacterize malicious behaviors, so this paper has extracted some subgraphsfrom the original dependency graph by key strings to analyze some typicalbehaviors of the malware.