Malicious Code Detection And Containment Technology

Computer security has become a major area in academic and industry, since the Morris worm broke out in 1980's. Malware, both in terms of attack and defense, is a fast growing field. Owing to Internet as well as homogeneity of software and hardware, people attach more importance to security issues. As one major form of computer attacks, malware poses great challenges to Internet and users, causing huge losses. However, there are still lots of unsolved problems, because existing detection approaches is ineffective, being incapable of dealing with diverse and sophisticated morphing attacks. False positives and false negative have greatly limited the use of detection systems. Meanwhile, traditional defense architecture is weak against outbreaking, large-scale attacks. Trojan horses are a typical class of host-based malware that aim to steal sensitive data, and they account for the largest proportion of malware classes; network-based malware is represented by worms, which propagate in a distributed fashion, hence it has unforeseeable potential threat, and may give rise to subsequent attacks. Therefore, they both deserve deep research and attention.This thesis surveys the principles, key techniques and detection approaches of malware. We focus on Trojans, viruses and worms. From the perspective of detection and containment, we have the following three contributions:1. We researched and implemented a Trojan/rootkit malware detection technique using virtual execution environment. We defined malware behaviors, security events of affecting operating systems. Data mining is used for detecting unknown samples. Virtual environment decreases the negative affects the malware brings to users. Experiment results showed that compared with our counterparts, our detection approach can detect unknown malware precisely.2. Due to the prevalance of morphing worms, the core technique of misuse detection, signature-based detection has been a bottleneck. Traditional signature generation technique takes a long time to generate as well as having high false positives. We proposed an accurate, effective signature generation technique which can resist morphing attacks, and for the first time, we apply it to viruses and Trojans. Initial experiments showed it can generate high-quality signatures in a short time with low false positives and low negatives, and it also gives quantitative analysis.3. In order to contain outbreaking, large-scale attacks caused by malware, we proposed a new distributed malware response and containment framework. It incorporates behavior-based anomaly detection and signature-based misuse detection. So it can not only detect known malware instances but unknown as well. We designed and implemented the prototype system. This framework considerably reduces human labor; hence it can respond quickly to outbreaking malware. This work is an initial attempt for containing large-scale attacks.