Technology Research Of System Log Anomaly Detection Based On Deep Learning

System log is widely used in various software systems.It can record system operating status information to help administrators debug and analyze software systems.In the field of system security,analyzing and mining the logs generated by the system to detect the abnormal behavior patterns of system is a popular security protection method.However,with the rapid development of computer software and hardware,the scale of software system and log data generated by system is becoming larger.Therefore,detecting anomly of system by manualy labeling on massive log data sets has become no longer realistic.In recent years,there are many related researches on log anomaly detection with different schemes.The utilization of deep learning neural network models for log anomaly detection has gradually become the popular approaches.However,the complexity of the system behavior pattern and the update iteration of the system reduces the practicability and effectiveness of these deep learning approaches.This thesis improves and optimizes the problems existing in the application of deep learning to system log anomaly detection.The specific research contents are as follows:(1)Detection scheme based on log template topic features(TFAD): In this scheme,we improve the preprocessing part of log template data,and propose a log template topic classification model based on log template toptic.In this classification model,this approach introduced LDA topic model to implement topic classification for log templates according to topic feature of log template.Using the log template topic classification model to preprocess the log template topic classification can significantly improves the effectiveness of log anomaly detection.In the improved scheme ETFAD,this paper improves the accuracy of log anomaly detection by replacing LSTM model with GRU model and integrating with self-attion mechanism.Finally,a log anomaly detection scheme based on deep learning with higher detection accuracy and stronger robustness in new log template injection is proposed.(2)Detection scheme based on parallel GRU classification model(DGAD): Considering that most of the existing deep learning-based log anomaly detection methods just detect context feature anomaly or frequency feature anomaly of log sequence,this chapter proposes an anomly detection method by a dual parallel GRU classification model.In this scheme,we design a dual parallel GRU classification model to detect sequence feature anomaly and frequency feature anomaly at the same time.The model inputs log template sliding window with log template frequency vector at the same time,and outputs the prediction result according to the current input.The detection result is determined by whether the current log template is an element in predition result set of the model.Finally,a deep learning-based log anomaly detection scheme is proposed,which can simultaneously detect log sequence context features and frequency features.(3)Based on the TFAD solution,we designed and implemented the prototype system from the bottom-up and modularized by analyzing the design requirements of the target system.And we conduct experimental comparison and analysis on the existing real data set and the existing similar mainstream schemes to prove that our proposed scheme can achieve higher accuracy and stronger log anomaly detection function.