Research On Trojan Horse Detection Technology Based On Communication Behavior Analysis

In recent years, malwares like Trojans have brought increasing threaten to network security. This thesis mainly studies the principle of Trojans' covert communication and the approaches of Trojan communication detection. In order to detect Trojans' communication behavior effectively and enhance network security, this thesis researches the Trojans' communication technology, analyzes Trojans' communication process and their behavioral characteristics. It establishes the model of the Trojans' communication concealment and the one of Trojans' communication behavior detection. Combining with the C4.5 decision tree classification algorithm of the data mining technology, it presents a highly universal method which can detects Trojans' communication behavior. And then a Trojans' communication behavior detection system with lower rate of false positive and false negative is designed and implemented.This thesis first introduces the concepts related to Trojans, the working principle and key technologies of the Trojans. Among the concepts, it focuses on the analysis of the Trojan's communication technologies. The mathematical model of Trojans' communication concealment is established according to the Trojan communication principle. Then, the concealment of practical samples' communication is analyzed with the concealment model. According to the analysis, it is concluded that Trojans cannot simulate the normal network's communication. The conclusion is also pointed out that behavioral-based detection technology is absolutely feasible. Still, it is revealed that the Trojans' communication concealment is related with the protocol concealment, the feature concealment, the traffic concealment and the behavior concealment. Then the quantitative evaluation criterion of the Trojans'communication concealment is proposed. The results of quantitative analysis are given.Secondly, the communication process is analyzed. The behavioral characteristic in each period is summarized. Meanwhile a Trojans' communication behavior detection model is established and a communication behavioral characteristic-based detection method is proposed on the base of communication behavior analysis. After that the flow of TCP protocol data is adopted as the layer of behavioral characteristics extraction referring to the method of network traffic analysis and Trojans' communication characteristics. And C4.5 decision tree is selected as the classification algorithm by contrasting classification algorithms of the data mining techniques.Thirdly, different behavioral characteristics are extracted according to the communication behavior of different periods. The distinguishing capacity of the characteristics is analyzed by combining time-frequency analysis technique. Six high-performance characteristics of which computational complexity does not exceed the linear level are given. And then the computational complexity of the method is analyzed to prove that the proposed detection method is effective. A system of Trojans' communication behavior detection is designed and implemented according to the detection method. In order to improve detection efficiency, an algorithm with rapid rate of processing called array-list-based algorithm is proposed. This system is proved to be practical and effective through the experiments. The result shows that the system can detect the various kinds of common Trojan and mutation Trojan.Finally, the ways to improve the system are pointed out.