Design And Implementation Of Anomaly Detection System Based On Log Analysis

Distributed systems are used as the underlying core architecture by more and more enterprises due to their high performance,scalability,fault tolerance and many other advantages.However,the workload and complexity required to maintain a distributed system composed of many components deployed in various places are quite huge,which makes manual operation and maintenance difficult to meet the demand,and an anomaly detection system needs to be introduced.Logs,as important data for recording the operating status of system components,can reflect the operating status of each component of the system.By analyzing the logs,it can effectively help operation and maintenance personnel to find abnormalities in the operating status of the system.Therefore,anomaly detection system based on log analysis has become one of the hot spots in the field of intelligence operation.This thesis designs and implements an anomaly detection system based on log analysis.The system collects the log data of each component of the distributed system,then stores and analyzes it,and provides real-time log abnormality detection and abnormal alarm functions.Aiming at the problem of semantic loss in the template extraction process,a semantic-based template parsing method is proposed.After the log segmentation is completed,a semantic repair step is added to ensure the integrity of the information.At the same time,the templates are clustered again after templates are generated by the log clustering.Merging redundant information to ensure the accuracy of the original semantics.In view of the impact of different contexts on the meaning of words and logs,a context-based feature extraction method is proposed,using a word granularity encoder and a sentence granularity encoder to encode log entries in different level,and using multi-head attention mechanism to capture context and provide accurate extraction of log characteristics in different contexts.In view of the timing characteristic and spatial calling characteristic of distributed system logs,a time-space-based anomaly detection method is proposed.Convolutional Neural Network(CNN)is used to focus on the call situations between logs,Attention Mechanism is used to focus on remote procedure calls with a larger span,and Long Short-Term Memory(LSTM)is used to focus on the timing characteristics of logs to improve the accuracy of anomaly detection.Firstly,this thesis introduces the research background of anomaly detection system based on log analysis,investigates related technologies and products,analyzes the requirements of an anomaly detection system based on log analysis,and conducts a series of research on log analysis,feature extraction and anomaly detection.Then the overall architecture design,module design and implementation of the anomaly detection system are introduced in detail.Finally,the effectiveness of the system is verified through a series of tests.