Design And Implementation Of Network Traffic Anomaly Detection System Based On Spark

With the development of the times,Internet technology has entered a high-speed development stage,and the network has become an indispensable part of people’s daily life and production.At the same time,network security issues have also become the focus of attention.The increasing complexity of network structure and the variety of network attacks lead to frequent network security incidents,which seriously affect economic development and social progress.At present,although there are various security products such as intrusion prevention devices and firewalls,most of them have certain limitations.For example,the intrusion detection system based on pattern matching adopted by industry cannot detect accurately in real time when faced with a large-scale high-speed network environment.Based on this,this paper designs and implements an efficient network traffic anomaly detection system to better cope with the increasingly complex network environment.This paper first expounds the background of the project,and further clarifies the significance of the project by comparing the research status in this field at home and abroad.Then,it gives a brief overview of the algorithm principle,development framework and other related technical theories involved in the project,and makes a detailed demand analysis of the project,so as to clarify the research and development objectives of the system.Then,according to the results of demand analysis,the overall design and detailed design of the system are carried out,and the overall system is divided into six functional modules: traffic collection,anomaly detection,threat analysis,threat report,intrusion retrieval and system setup,and each module involves several sub-functions.In terms of research and development technology,the whole project is developed by using Spring Boot framework,using Spark distributed computing framework based on memory computing to realize parallel construction of models and distributed intrusion detection,using kafka message queue to realize efficient transmission of traffic information,and using Redis and My SQL storage systems to store data.In anomaly detection,firstly,a one-dimensional convolutional neural network based on Le Net framework is constructed to extract traffic characteristics,and then the intrusion detection model is designed by combining with Light GBM classification algorithm.After the development is completed,the function of the whole system is tested by designing test cases,and the performance of the system is tested by combining test tools to ensure that the results meet the expected requirements.Finally,the process of software development and paper writing is summarized,and the future development direction of the project is prospected.Since it was put into use,the traffic anomaly detection system has been running smoothly,which can meet the requirements of near real-time processing and detection of large-scale network data,and provide solutions according to different types of network attacks,improve the emergency response ability of network managers,reduce the harm caused by abnormal networks,and further ensure network security.