Design And Implementation Of Network Security Traffic Analysis And Perception System Based On Big Data

With the booming development of the Internet and the advent of the 5G era,the network has become more and more integrated into people's daily life,and network security has become the focus of social attention.Facing the current complex network environment,the application of big data and machine learning technology in the field of network security has become the focus of industry research.This thesis designs and implements a network security traffic analysis and perception system based on big data,which has two core functions:network security traffic analysis and early warning,and network attack tracing.Starting from the core functions,this thesis has completed the following works:1)network traffic data acquisition and processing module,which supports cross-platform work,collects and analyzes data packets based on Gopacket technology,obtains 84-dimensional feature vectors and traffic five-tuple information,and passes the data information to the data storage module;2)data storage module,which is based on Kafka,cuckoo filters,mysql and Redis to design and implement three sets of storage solutions:K storage,C storage,and D storage.It can meet the needs of different modules for real-time data transmission,efficient data storage,and random query,and solves the problem that a single traditional database cannot well adapt to large data and high concurrency,which has certain practical value;3)abnormal traffic analysis and warning module.In this thesis,streaming k-means is used for abnormal traffic analysis and warning based on Spark.The advantage of the model is that data annotation is not required,and 74.35%recall rate is obtained on CICIDS2017 data set.4)attack tracing module,this thesis discusses and implements the two kinds of algorithms:master-slave attack tracing and recursion attack tracing,and uses the OPNET software to build a semi-physical simulation environment to test the functionality and performance of the attack tracing module.After 20 minutes of simulation,2000 queries were made on average,and the two algorithms achieved a success rate of 98.95%and 97.85%,respectively.The results show that master-slave attack tracing algorithm can better solve the attack tracing problem and has certain practical value.5)interactive query module,which is the back-end of the service,contains data service and data management functions,and provides service interface and data support.At the end of this thesis,the system was tested for function and performance.The test results verified that each module of the system could coordinate with each other,confirmed the effectiveness and reliability of the system in abnormal traffic analysis and network attack tracing,and had certain application value in improving network security under the condition of big data.