Research Of Web Log Abnormal Traffic Detection Technology Based On Machine Learning

At present,with the rapid development of Internet technology,the problem of network security is more and more serious.The Web log contains the record of information and operations of users.By analyzing the log,developers can detect abnormal network traffic and find vulnerabilities in time.Because the traditional log analysis technology is based on rules and pattern matching,The processing of massive data is not good,and it is unable to deal with complex and diverse network attacks.With the rise of artificial intelligence,it is popular to apply machine learning and deep learning technology to log analysis.This paper proposed a method of custom features.Compared with other statistical information-based feature extraction,this method focuses on the URL part of feature extraction,ignoring other information that is redundant and does not improve the detection result.It is effective to reduce the dimension of the feature which is extracted from log data.The result proved that the detection result of this feature extraction method is good,and the detection effect is slightly higher than other feature extraction methods based on statistical features,but the feature dimension was greatly reduced.Then,this paper presented a method of feature extraction and studies a method of feature coefficient adjustment alogrithm related to distance measurement,which was applied to KNN alogrithm and improved the accuracy and precision and other index of anomaly detection based on server log.The main contents are as follows:Firstly,this paper introduced the structure and characteristics of Web log,and introduced two methods of Web log abnormal traffic detection,misuse detection and anomaly detection.What is more,anomaly detection can be subdivided into machine learning algorithm and deep learning algorithm based anomaly detection,and then introduces the related algorithmsThen a feature extraction method based on statistical knowledge was used to extract features from Web logs.Compared with some other feature extraction methods,the detection effect is relatively good,but the dimension of feature extraction was far lower than other methods.After further dimensionality reduction means,At the expense of smaller detection results,in exchange for a further reduction in dimensionality,which greatly saved the storage space of data features and improves the processing speed of subsequent algorithms.Next,This paper put forward an algorithm named feature coefficient adjustment,which is different from the traditional feature engineering,such as feature standardization and feature screening.Feature coefficient adjustment focuses on the difference of importance between different features,and gives different weight coefficients to adjust the contribution in distance measurement.Then particle swarm optimization algorithm was used to make the adjustment of the characteristic coefficient adaptive.The results showed that the accuracy is improved by 0.84% and the accuracy is improved by 1.00% and f1-score increased by 1.09%.Finally,this paper established a log system based on ELK,and carried out experiments and simulations overall in which the related technologies involved in the previous chapter and analyzed the results in detail.