Research On Intrusion Detection Technology For Large-scale Networks Based On Hadoop

The rapid development of Internet and Computer is be for the whole society,but also increase serious security problem,so Intrusion Detection Technology is very important.In the Big Data and Internet of things,the network scale is increasing rapidly,how to find out the illegal and intrusion behavior becomes a hot research.This paper firstly introduces the related knowledge of Cloud Computing and Hadoop,and points out the huge advantage to process and analysis data in Big Data.Combined with the Intrusion Detection Technology,researching the architecture of Lightweight Intrusion Detection System Snort,To analysis of large scale network intrusion detection problems.Here are three things:(1)It introduces the applications of fuzzy integraltheory in intrusion detection alert fusion,also analysis of the Support Vector Machine algorithm.After this paper analysis Intrusion Detection Technology and Cloud Computing careful,combined with the characteristics of large-scale network environment,it puts forward a kind of large-scale network intrusion detection method based on Hadoop,this method mainly includes two parts:Distributed detection data packets with Snort and processing the alert logs.(2)In large-scale network environment,the data flow is very large,this paper deployed the Snort with Hadoop cluster,and process data packets with MapReduce,then using fuzzy integral method to fusing the alerts.Experiments show that the method is available and effective.(3)For the problems on storage space and response time with the rapid growth of network security data scale, we proposed I-MSMO, a parallel SMO algorithm based on MapReduce. Experiments show time complexity of the proposed I-MSMO algorithm reduced obviously, with keeping high detection rate. At last, the variation of time and space complexity of the proposed algorithm is analyzed to confirm the effectiveness.(4)This paper laid the foundation of technology to further expand huge amounts of data anal laid the foundation of technology.