Research On DNS Tunnel Trojan Detection Technology Based On Communicaition Behavior Analysis

DNS(Domain Name System)is a distributed database that maps the domain name and IP address,which is one of the most important infrastructures of the Internet.Firewall,security software and other general security policies would not block the DNS message,which creates a unique advantage for the construction of the convert tunnel based on the advantages of DNS protocol.Initially,the original intention of the DNS tunnel tool is to skip the Internet login authentication and free access the Internet.Howerver,DNS tunnel is gradually being used by some new Trojan as a covert remote control of the channel,which is a serious threat to network security.This paper proposes a DNS tunnel detection method based on the analysis of communication.It first analyzes the communication behavior of DNS tunnel Trojan from the perspective of DNS conversation.Second,it extracts eight features between DNS tunnel Trojan sessions and the normal DNS sessions which is composed of DNS session evaluation vector.And then it builds classifiers using the improved random forest decision algorithm.Finally,a DNS tunnel detection model based on communication behavior analysis is constructed.Experimental results show that the model is practical and effective.The main work and contributions of this thesis are as follows:Firstly,the current DNS tunnel technology is analyzed and summarized.And then the operational principles and the communication technique of DNS tunnel Trojan are studied.The communication process of DNS tunnel Trojan is divided into two main stages,including on-line request stage and interactive transmission stage.With the theory of finite state machine,a state transition model of DNS tunnel Trojan's communication behavior is proposed.Secondly,a mothod of the DNS tunnel Trojan horse detection based on the communication behavior analysis is proposed.Aiming at the problem that the traditional DNS detection method based on load detection and flow monitoring has a high false alarm rate and can not effectively detect the problem of the new DNS tunnel Trojan,this paper presents a new detection method based on the communication behavior analysis.It first analyzes the communication behavior of DNS tunnel Trojan from the perspective of DNS conversation.And then it extracts eight features between DNS tunnel Trojan sessions and the normal DNS sessions which is composed of DNS session evaluation vector.At last,it generates a set of behavioral attributes with good classification effect and strong universality,and extractes DNS session real time evaluation vector and DNS session alternative evaluation vector.Thirdly,a decision model based on improved random forest classification algorithm is proposed.Aiming at the problems existing in the traditional random forest algorithm for imbalanced data sets,we propose an improved random forest algorithm based on negative class center distance.And the improved random forest algorithm is tested and evaluated in terms of classification performance and generalization error.Finally,combined with extracted DNS evaluation vector,the DNS tunnel Trojan detection training model based on improved random forest algorithm is established.Finally,a DNS tunnel Trojan detection system based on the communication behavior analysis is designed and implemented,and the validity and practicability of the system was tested.The model includes four parts: data packet acquisition and integration module,DNS session reorganization module,random forest classification training module and DNS tunnel Trojan horse traffic detection module.Experimental results show that the method of the DNS tunnel Trojan detection based on the communication behavior analysis can not only detect the high hidden DNS tunnel Trojans effectively with low false alarm rate and false negative rate,but also has high detection ability of unknown DNS tunnel Trojan,which means that the this method is practical and effective.