Research On Network Intrusion Method Based On Machine Learning And Big Data Technology

With the popularity of computer networks,people develop and use multiple applications to run in computer networks,but cyber attackers try to use the weak links of the network architecture to steal and destroy valuable information.In order to maintain a good network environment and protect people's information,network security issues need to be resolved.So Network intrusion detection technology,especially anomaly detection technology,has become one of the most influential research directions in the field of network security.The essence of intrusion detection technology is the process of monitoring and detecting malicious attacks in network protocols to discover abnormal network behavior.At present,the network intrusion detection technology based on artificial intelligence or machine learning still has the problems of low detection rate,long time consumption and small amount of processed data.Therefore,this paper improves the detection accuracy and the ability to process big data by improving machine learning algorithms and network intrusion detection methods based on big data technology.The main research work of this article includes:(1)In view of the problems of machine learning in network intrusion detection,the performance of intrusion detection is improved by analyzing the structure of the neural network.Based on the research of the RNN neural network and LSTM neural network structure,this paper proposes the GRU-SVM neural network algorithm.The GRU neural network not only solves the problems of RNN long-term memory and gradient,but also contains only update and reset gates in the structure.The gating is simplified,and the calculation efficiency is improved.The experimental results verify that the GRU-SVM algorithm has significant advantages over the traditional algorithm in detection rate and training time.(2)Aiming at the problem that traditional machine learning algorithms cannot handle big data,introduce big data technology,and on the framework of Hadoop and Spark platforms,design a distributed feature selection model based on mutual information algorithms,which includes column transformation and calculation-related There are five parts:performance,computing redundancy,histogram creation and mutual information calculation.The model design conforms to the structure of the Spark distributed computing framework.This model solves the problems of low detection rate and long time consumption caused by too high data dimensions in intrusion detection technology.Combined with the advantages of Spark distributed computing framework,the amount of data processed is up to about 100G.The experiments are compared in SVM,Naive Bayes and decision tree algorithms.The experimental results verify that the distributed feature selection method based on mutual information proposed in this paper can extract features with high correlation and improve classification than the principal component analysis algorithm.The detection accuracy of the algorithm.The two algorithms proposed in this paper are tested in the UNSW-NB15 data set and the NSL-KDD data set respectively.Compared with other classic algorithms,this algorithm has a high detection rate and a large amount of data.Combined with big data technology,it expands the network technical scope of intrusion detection.