Research On Anomaly Log Detection Method Combining Time And Semantic Information

In recent years,with the increase in the complexity and scale of software systems,logs have gradually become important data for effectively capturing,analyzing,and lo-cating anomalies in system runtime.Anomaly detection based on log data is one of the most important methods to achieve auxiliary system maintenance,and this method aims to effectively perform system anomaly mining through log analysis automatically.In the field of log-based system anomaly diagnosis,log parsing and anomaly de-tection methods are the key contents of them.Among them,log parsing is a process of extracting log templates from original logs.The current mainstream log parsing methods rely too much on regular expressions to process logs,which cannot effectively cope with the increasing complexity of log structures as technology advances and iterates,and the portability of this approach is low.In terms of log anomaly detection,the current main-stream approaches mainly focus on the discovery of execution path anomaly,ignoring the feedback of system anomaly in log output delay and also ignoring the correlation between different anomaly types.To address these issues,the main research of the thesis includes the following aspects:(1)Aiming at the problem that current mainstream log parsing methods cannot effec-tively parse logs with large complexity,the thesis proposes a new online parsing method for logs based on the tree structure.During the template finding process,the method achieves multi-layer division of log data through a fixed depth tree structure to reduce the number of logs and templates compared in the log parsing process.During the tem-plate matching process,the thesis independently optimizes the log template structure by merging similar templates to further improve the accuracy of log parsing.(2)Aiming at the problem that the current mainstream anomaly log detection methods have incomplete and imprecise coverage of anomalies,the thesis proposes an anomaly log detection method based on TCN-attention and multi-task learning by combining the time and semantic information of logs.Based on the log template semantic vector and the output time interval of the log,the method creates the task of execution anomaly detection task and the task of delayed anomaly detection.With the advantage of multi-task learning to share shallow parameters,the method can realize different anomaly types to correlate with each other during training,thus improving the detection effect of the anomaly log detection method.(3)The thesis designs related experiments based on the public log dataset and con-ducts a comprehensive evaluation of the research in the thesis.The experimental results show that: In terms of log parsing,compared with other mainstream log parsing methods,the method in the thesis is more suitable for complex log structures and can more accu-rately extract log templates? In terms of anomaly detection,compared with other main-stream anomaly detection methods,the anomaly log detection method in the thesis has a certain improvement in recall rate and precision rate,and achieves the precision rate of97.2% and the recall rate of 94.1% in the HDFS log data set.