Research On Systems Logs Analysis And Anomaly Detection

With continuous development of the internet of things industry,scale of systems continue to expand,which put forward higher requirements for anomaly detection.Logs record condition of systems running,which are important resources for developers to maintain systems.However,anomaly detection based on logs face many challenges:massive and semi-structured logs,irregular updating of log statements,high requirements for real-time and accuracy of anomaly detection.Therefore,this thesis focuses on online logs parsing and anomaly detection.The specific research contents are as follows:1.By analyzing characteristics of logs,this thesis proposes an online logs parsing method based on prefix tree(OLMPT).Firstly,OLMPT preprocess logs by filter and segment correctly.Secondly,OLMPT extracts the first letter of logs templates to form strings,constructs a prefix parsing tree,and saves templates information of logs in each leaf node.Thirdly,strings of new logs are obtained and compared with strings in logs templates.Finally,the parse tree is updated in real time.Compared with existing three logs parsing methods,it is proved that OLMPT not only has higher parsing accuracy on single source and multi-source logs sets,but also has shorter parsing time on logs sets,which can better meet the needs of online logs parsing.2.Based on OLMPT,this thesis proposes an anomaly detection method based on Transformer encoder(Log Transformer).Log Transformer can make good use of characteristics of context information to encode logs templates,and use spherical loss function to classify logs.Through experiments on three logs sets with anomaly labels,it is proved that Log Transformer is better than existing anomaly detection methods,which based on logs templates indexes and logs templates semantics.In addition,this thesis analyzes the impact of different parsing methods on anomaly detection.Experiment shows that the accuracy of Log Transformer is higher than anomaly detection methods which based on log templates indexes.Aimed at perfecting existing logs parsing and anomaly detection methods,and improving the effect of anomaly detection,this thesis proposes a general method of logs parsing and anomaly detection for different systems,and verifies the applicability of the method through comparative experiments.