Research On Anomaly Detection Based On Clustering Technique

With the rapid development of computer technology and network communicationtechnology, and the widespread application of computer, more and more security problems ofnetwork have been disclosed. Traditional protection measures such as firewall system and dataencryption can hardly satisfy the network security requirements nowadays. Anomaly detection isa way to detect intrusion. Compared with traditional security technologies, it’s a proactivedefense technology which can react according to the intrusion data detected before the attackagainst the network system. The key point of anomaly detection technology is to adjustdynamically in accord with the change of user’s behavior, so as to lower the false alarm rate anddetect the unknown intrusion. However, the gradual increase of different intrusion measures,along with the massive growth of network data, brings enormous pressure to the intrusiondetection system. The concept of data mining provides effective technological solution for theintrusion detection. But the past intrusion detection technology based on data mining need thedata to be marked, and requires the "purity" of data samples. Clustering analysis technology, as alearning approach without supervision, can build model on the unmarked dataset and find theanomalous data, thus overcoming the defects brought about by the traditional data miningapproach.Based on the background above, depended on the "High reliable network servicemanagement system" of State863projects, this paper studies the anomaly detection withclustering technology. According to the high false alarm rate of the current anomaly detectionsystem and its inability to meet the requirement of real-time capability, this paper puts out animproved network anomaly detection model based on Hi-WAP clustering, then proposes theHi-WAP(Hierarchical-Weighing Affinity Propagation) algorithm by adjusting the core algorithm.As feature selection is brought into anomaly detection system, a KA-APC(Kernel-basedAdaptation-Affinity Propagation Clustering) feature selection algorithm comes into being. Animplementation method of network anomaly detection system based on feature selection andclustering is designed. Details are as follows:1) First, the traditional network anomaly detection model based on clustering is improved,and a Hi-WAP core algorithm is proposed. Faced with data attack on a large scale in training period, the algorithm of anomaly detection model based on clustering cannotkeep the accuracy and speed on a high level. The improved model selects the propertyin the pretreatment of data, so the data can be directly clustered. Hi-WAP algorithmcarries out AP clustering on the data which is hierarchically weighted and is put out ofclustering centre later. The experiment of the improved model and algorithm with thehelp of KDD CUP99dataset shows a rather good result that, compared with traditionalK-means algorithm, the average detection rate is increased by9.2%, average falsealarm rate reduced by1.57%, and the time for detection shortened by10.11s.2) A KA-APC feature selection algorithm is proposed. As the intrusion data has thecharacters of high dimension and large redundancy, this algorithm depicts the profile ofintrusion data by manifold searching approach, calculates the similarity matrix of thespace by casting the data searched out by manifolds to high dimensional space, usingkernel mapping approach, then clusters it with AP algorithm principle and output thefeature subset. The result of the simulation shows: this algorithm can effectively reducethe dimensions of network intrusion data and increase the classification accuracy offeature selection. Compared the traditional K-means algorithm, the selected propertiesare reduced by4.2averagely, and the accuracy of classification improved by11.9%.3) The implementation method of anomaly detection system based on feature selection andclustering is designed. The detection rate and false alarm rate are tested to see theperformance of the system. And the result shows that this anomaly detection systemcan effectively detect the intrusion attack under the network environment of hundredtrillion, with an improved detection rate, which can even reach92.1%, and a relativelylow false alarm rate. Compared with the traditional clustering-based anomaly detectionsystem, this is a great improvement.