Research And Implementation Of Intrusion Detection System Based On Spark

With the rapid improvement of equipment performance and communication technology,the Internet is deeply integrated with people's life,which brings convenience.However,the problem of network security is becoming increasingly serious.Network security has a far-reaching impact on society,economy,politics and military.It is the cornerstone of national security,and intrusion detection system is one of the important means to deal with network security problems.The traditional detection system based on misuse detection algorithm uses pattern matching method to detect attack events.Although it has the characteristics of low false alarm rate and high speed,it needs to specify the attack behavior in advance,has no ability to detect new attacks,has high omission rate,and needs a lot of expert experience.With the development of artificial intelligence,the detection system using artificial intelligence algorithm not only has high detection accuracy,but also has the ability to adapt to new attacks through the feature matching detection method,which has become a hot spot of current development and research.This paper studies the existing artificial intelligence algorithm,designs a suitable intrusion detection model for the system,and uses big data technology spark to realize the system,so that the system has the ability of parallel computing.The main work and innovation of this paper are as follows:(1)The system collects web logs in real time and detects SQL and XSS injection attacks through web logs.Aiming at the design of Web log detection model,in order to make the model take into account the detection accuracy and detection efficiency.Use Git Hub open source web log data and CSIC2010 data set.Compare and analyze the detection effects of artificial feature extraction and TF-IDF feature extraction methods combined with five common traditional machine learning algorithms on accuracy and efficiency.The artificial feature extraction method combined with the XGBoost algorithm is selected as the detection model for Web log detection.In order to avoid artificial feature extraction containing redundant features,the m RMRXGBoost feature selection algorithm achieves better results than the single feature selection method,reduces feature dimensions and further improves the accuracy of model detection.(2)The system monitors the network traffic data in real time,and detects Do S attack,brute force cracking,port scanning and other intrusion behaviors according to the network traffic.Aiming at the model design of system network traffic identification,because the unknown attack outside the training sample is one of the main reasons for false positives and missing positives,in order to make the model be able to identify the unknown attack outside the training sample.Using the CICIDS2017 data set,this paper improves the Le Net network model and adds the confidence output,so that the model can effectively identify unknown attacks on the premise of high classification accuracy of known attacks.(3)In order to avoid the performance bottleneck problem in the case of large-scale data concurrency,spark is introduced to implement intrusion detection system.The features of spark distributed computing and memory based iterative computing are used to improve the system's ability to deal with large-scale data,and at the same time,the real-time processing ability of spark is used to meet the real-time requirements of intrusion detection.