Design And Implementation Of Anomaly Detection System Based On Machine Learning

With the continuous development of information technology,computers bring convenience to people.At the same time,network attacks from outside and abnormal events from inside emerge endlessly,which brings severe challenges to the detection of security anomalies.Traditional anomaly detection technology intercepts anomaly events by maintaining rule base.This hard coding technology lacks the adaptability to the emerging new security anomalies,and needs to upgrade the system by regularly updating the rule base.To solve these problems,this paper designs an anomaly detection system based on machine learning.Using machine learning model to discover anomalies by analyzing the network traffic data of the system.Because many network attacks,such as directory traversal attacks,are directly reflected in URLs.So the main scenario of this system is anomaly URL detection.In the aspect of model building,machine learning model is constructed by analyzing historical access data.TF-IDF is used for feature extraction,and feature dimension reduction is achieved by Autoencoder and Kmeans,which significantly improves the accuracy and time efficiency of model.In the aspect of system construction,the detection system is divided into several microservices.Message queues are used for communication between microservices,which reduces the coupling between the key nodes in the process,and makes the system have better expansibility.The new data generated in the process of detection can be used to upgrade the model iteratively,so that the system has better adaptability.In addition,the system provides a friendly human-computer interaction interface,users can easily participate in the process of anomaly detection.This paper first introduces the related technologies such as TF-IDF,Autoencoder,message queue,microservice and so on.By comparing with the existing network anomaly detection methods,this paper proposes a detection system.Requirements analysis is carried out based on system objectives,and then the concrete realization of each service in the system is illustrated,including communication based on message queue,construction of pattern cache,design of pattern elimination algorithm,etc.Then,this paper designs an anomaly detection algorithm based on machine learning,focusing on feature extraction and feature dimension reduction processing.Finally,through real network dataset,the accuracy of model and the efficiency of system are tested,which verifies the advantages of the detection system proposed in this paper.