Research On System Anomaly Detection Based On Log Information

With the rapid development of information technology,computer applications have penetrated into every aspect of people's life.The logs of a computer system record the status information and important events at runtime.Therefore,the logs can reveal the system performance problems and functional failures,and help the engineers to conduct in-depth analysis on the root of the problem.At present,researchers in this field have developed a series of anomaly detection solutions based on log information,but the current solutions have the following problems: error in log parsing,feature extraction limited to sliding window perception domain,poor detection performance with small size of abnormal samples,and etc.Based on the above background,this thesis conducts the following research:1)A log anomaly detection algorithm based on hierarchical Transformer model is proposed.For the scenario with sufficient abnormal samples,a model based on supervised learning framework is designed to fully mine the abnormal patterns of samples.This model has a hierarchical feature extraction structure,which solves the problem that feature extraction of logs is limited by sliding window perception domain.On the other hand,traditional anomaly detection schemes need to use log parsers to obtain log templates.In this algorithm,this step is replaced by directly obtaining the embedded representation of a single log through deep learning language model,so as to avoid the error accumulation caused by log parsers.2)A self-labeling algorithm based on confidence information is proposed.The training set in this algorithm is composed of only normal logs,so it is very friendly to the scenarios where abnormal samples are difficult to obtain.Compared with the model based on supervised learning framework,this algorithm has stronger flexibility and universality.On the other hand,this algorithm subtly transforms the one-class classification problem into multi-class OOD problem,which effectively alleviates the loose boundary phenomenon under the definition of single classification problem,and the experiments prove that this algorithm can further improve the anomaly detection result.3)A log anomaly detection system based on deep learning algorithms is designed.By integrating the above two algorithms,the system can not only show excellent detection performance in the case of sufficient abnormal data,but also maintain good performance in the case of insufficient abnormal samples.In addition,the system meets the multidimensional requirements of the practical production scenarios,including user management,model management,data management and etc.At last,the author conducts abundant experiments and the results prove that the system has excellent performance.In general,this thesis designs two log anomaly detection algorithms for different scenarios,and the result of experiments prove that these two algorithms have achieved SOTA results in their respective scenarios.Finally,an anomaly detection system integrated with the above algorithms is designed and implemented,and the system shows excellent detection ability and universality in the experiments of five different production scenarios.