| Intrusion detection is a defense method to ensure network security.The traditional intrusion detection method has unsatisfactory detection ability and response speed when dealing with massive network data.Distributed intrusion detection system integrates computing and storage resources of several hosts,which has improved detection ability to a certain extent,but not good at real-time data processing.In order to improve the capability of intrusion detection and real-time data processing,this thesis studies and researches the distributed platform and machine learning algorithm,and modified relevant algorithms for intrusion detection.The main work contents include:Aiming at the problems of large amount of intrusion data,decreasing detection ability of traditional algorithm and large time cost,combined with particle swarm optimization algorithm,k-means is improved,and PSO algorithm is introduced into the improved algorithm to comprehensively analyze the change relationship between local and overall optimal particle,and optimality analysis is carried out in the processing of each batch of data.Ensure optimal results at each step to improve detection capability.Through the experimental comparison of different intrusion detection data sets,it is proved that the improved algorithm in this article reduces the time cost and significantly improves the processing capacity compared with the unimproved algorithm.Aiming at the problems of high delay and untimely data distribution of traditional distributed platform in the field of intrusion detection,an intrusion detection method based on distributed real-time data analysis is proposed.Under Spark platform,firstly optimize file monitoring and resource allocation during data collection,and then improve resource allocation according to the corresponding relationship between Kafka theme and partition to ensure as few idle resources as possible,improve data distribution efficiency and ensure real-time performance.Finally,the improved clustering algorithm is used under the memory-based calculation method.The waiting time from data acquisition to processing is reduced and the real-time data processing capability of the detection method is further improved.Through the comparative test of KDD data,the results show that the improved algorithm in this thesis has improved in different amplitude in each evaluation index,and has more real-time processing ability. |
