| In recent years,with the popularity of network applications and the iteration of technology,more and more people rely on the internet for work and daily activities.However,the increased reliance also brings out a growing concern for network security.Various machine learning or deep learning are widely used in network security anomaly detection.Combined with the classification task of machine learning to detect user traffic,it can effectively determine the type of user traffic anomaly.At the same time,how to reduce the delay of anomaly detection response is also an important aspect of the Internet network.An important research direction of security.Based on the reasons above,this thesis focuses on the development and design of a user anomaly detection platform.The platform uses CICFlow Meter as the feature extraction of the original traffic data packets,builds a machine learning classification model to classify the feature data,and performs real-time statistics and data retrieval based on the classified data.The system has been enhanced to design the corresponding functional requirements.In the meantime,according to the non-functional requirements of the system,the corresponding functional logic flow is improved.The work of this thesis can be outlined as follows:(1)Development work for real-time detection.In this thesis,the feature extraction CICFlow Meter and the machine learning classification detection module are added to the message queue,so that the feature extraction and anomaly detection can be logically operated in parallel to reduce the overall time of the detection process.At the same time,the detection and classification tasks do not need to wait for the overall calculation of the characteristics of the traffic data packet to be completed before classification Work,ensuring the overall timeliness of the system.(2)Real-time statistical work for abnormal data after classification is completed.This thesis uses the sliding window mechanism of the stream processing framework Flink to realize real-time calculation,and save the calculation results to the memory database.Therefore making the front-end page response data no longer perform corresponding calculation work to reduce the response time,avoid the corresponding pause time caused by massive data query calculations,and to maximize user experience.(3)It is aimed at the user’s query and retrieval work for abnormal type data.When the amount of abnormal data is too large and reaches more than one million levels,the corresponding query optimization of the traditional relational databases has little effect,and even due to factors such as field design and other related factors,the query cannot go through the index and needs to scan the entire table,etc.,resulting in pauses for users to query the page Too long or even query failure,seriously affecting user experience.This article uses the full-text search engine Elasticsearch,and uses its inverted index to perform fuzzy queries,which greatly reduces system pause time and effectively improves user experience. |
