Reserch On Real-time Detection Algorithm Of Network Intrusion Based On Spark

Tremendous data generated by modern Internet that with a rapid development nowadays and many kinds of privacy information is a part of it.The attacks on network internet have never been stopped and they are becoming more complex and globalized.Network intrusion is now the most common one among the types of anomaly intrusion.There is some vulnerability in the computer network,and the operating system neglects the security problem in the early stage and pays too much attention to the function,which makes people inevitably be threatened by the network abnormal intrusion when using the network.Intrusion detection comes into being as an important supplement to static security technology and becomes a significant part of network security.Both the internal unauthorized operations and the external intrusion will be monitored and tested.However,it is hardly possible for a single network intrusion detection model and detection method to achieve real-time network intrusion detection and timely response when it deals with massive and high-speed internet data,which significantly decreases the quality of intrusion detection.Based on the premise of network operation in big data system and the development trend of intrusion detection technology intelligence,this paper proposes a real-time network invasion detection model NRIDS based on Spark.The model uses the Spark Streaming stream processing module to receive,process,and return the data collected by Kafka in real time,and uses Spark's machine learning module to train the network intrusion detection model online so as to realize network intrusion detection in the face of massive network data.It can detect intrusion attacks in real time,quickly and reliably and respond in a timely manner.The work of this article mainly includes the following aspects:(1)According to the deployment model and general model of network anomaly detection,combined with the common intrusion detection model and the demand for distributed processing of a large amount of network data,a network invasion real-time detection model NRIDS model based on distributed computing platform Spark is proposed.This model uses the stream processing module and machine learning module in Spark to jointly complete the real-time detection task of big data.(2)According to the application scenarios of various algorithms in the machine learning module in Spark,a clustering algorithm is selected to train the intrusion detection model so as to realize the need of detecting unknown attacks.Through the analysis and research on the principles,defects and clustering evaluation criteria of the standard K-Means clustering algorithm and real-time K-Means clustering algorithm,the necessary research on the real-time clustering algorithm of KMeans combining Spark and clustering analysis is carried out.Improve.And use the KMeans real-time clustering algorithm as the detection model training algorithm of the NRIDS model to achieve real-time detection of large-scale network data.(3)According to the division of the technical architecture functions and the overall flow of the NRIDS model,the various modules of the intrusion detection model are designed and implemented in detail.Finally,the KDD CUP 99 data set was used for simulation testing.Through the analysis of the clustering results and test results,the detection effect of the real-time network intrusion detection algorithm based on Spark was evaluated and verified.The test results show that the Spark-based network intrusion real-time detection model designed in this paper can deal with the detection of large-scale network data and achieve real-time detection.At the same time,compared with the real-time KMeans clustering algorithm which is not optimized and the real-time KMEans clustering algorithm which is optimized by PSO optimization algorithm,the optimization of the algorithm improves the detection accuracy.