The Research Of Intrusion Detection Technology Based On Outlier Mining Under The Hadoop Cloud Platform

With the rapid development of information technology, especially the development of Internet technology, the network age is coming. Human beings enjoy the convenience network brings to them. However, the Internet is a double-edged sword. As well as its convenience, it brings some destabilizing factors. Users’ obsession with moderate losses caused by network security haunted all the time. Intrusion Detection Technology aims to all kinds of network attack and it is efficacious for solving problems caused by attack, which detects the intrusion by analyzing network packets. The data, which the system has to handle, however, becomes more complex. What’s more, the traditional detection system was unable to meet the demand of current intrusion detection because of the huge network flow. Thus, it is urgent to find out a way to make detection system adapt to the current network environment.According to the two data characteristics that intrusion detection system needs to deal with: high dimension and mass. This thesis presents a mining outlier algorithm based on attribute correlation and probability of outlier to detect intrusion behavior. The algorithm is for the purpose of improving the performance of intrusion detection system through combining the data mining to intrusion detection technology. Data mining is a technology which digs the useful information from a massive data, and outlier mining is a kind of data mining technology which finds abnormal data in data set. The aim of outlier mining is very consistent to the intrusion detection technology’s goal which is to detect abnormal behavior in all behavior. The first step of the algorithm proposed in this thesis is to get the subset of high dimensional attribute set by attribute correlation analysis and attribute reduction, which the subset of attribute can retain the important information of the original data set. Then the algorithm will calculate the outlier probability of the data set on the subset of attribute to detect intrusion behavior.Although through combining data mining technology to intrusion detection technology can be very good to make intrusion detection system to adapt to the current network environment. But the performance of the traditional centralized intrusion detection system is limited. Therefore, this thesis applies the algorithm to the cloud platform, in another words, this thesis uses the parallel of the algorithm to improve the performance of intrusion detection system. And Hadoop is an open source cloud platform currently widely used, which has the advantages of high reliability, good scalability, fault tolerance etc. Therefore, this thesis will be combined the algorithm with the principle of the MapReduce of Hadoop to improve the performance of intrusion detection system.Finally, this thesis carries on the experiment with the proposed algorithm and parallel algorithm using KDD CUP99 data set. The experiment results show that the algorithm can effectively detect intrusion behavior, and the performance of intrusion detection system is improved greatly.