The Design And Implementation Of Real-time Analysis System Of Network Security Log Based On Flink

With the explosive growth of the Internet,network security problems have become more severe.Compared with the previous network attacks,the current forms of network attacks are more complex and covert,which makes the detection of network attacks more and more difficult.In the context of the vigorous development of many Internet products,analyzing network security logs can well obtain the network security status of the server and can effectively detect whether it is subjected to network attacks.In order to monitor the network security status of the enterprise's product server,the internship company proposed the need to analyze the network security log generated by the product server to monitor the server for network attacks in real time.Ensure that the product server can run healthy and stable.This thesis designs and implements a realtime analysis system for network security logs based on Flink based on big data technology and cluster analysis methods.The system includes four modules,among which the log collection module realizes the distributed collection and unified storage of massive logs;The log preprocessing module implements the log cleaning function,the key feature digitization function and the feature value information entropy value conversion function.The log monitoring module implements log cluster analysis function and log real-time detection function.Among them,when implementing log clustering analysis,an analysis method based on the combination of information entropy value and K-means clustering algorithm is proposed.And according to the needs of the scene,the introduction of Davies-Bouldin index(DBI)and the maximum and minimum distance method have improved the process of K value selection and initial cluster center selection in the K-means algorithm,making the analysis process more accurate and applicable.The real-time detection part realizes the function of realtime detection of DDoS attacks through clustering models and Flink streaming computing methods;The alarm management module implements the log analysis result display function and alarm information feedback function.Finally,this thesis deploys the system in a cluster environment and performs functional and performance tests on the system as a whole.Test results show that the system can successfully complete network security log analysis and real-time detection and alarm of DDoS attacks,which meets the design goals and production requirements.