A System Log Anomaly Detection Method Based On RNN

In recent years,with the rise of online payment,e-commerce,instant messaging and other applications,large-scale distributed system has also been widely used,and is becoming the core component of the IT industry.Compared with the traditional single server,large-scale distributed system needs to guarantee its normal operation all day long,to provide reliable services for a large number of users.Any significant outage of such a system can result in an untold loss of revenue,so an exception detection framework needs to be built to ensure system reliability.In terms of providing system reliability,system logs can be well qualified for this task,so the exception security detection method based on system logs comes into being.Logs are widely used in system management to ensure reliability and are important data sources for detecting system anomalies.Before the abnormal detection process of the system log,it is necessary to parse the original text structure log into a structured log that can be recognized by the anomaly detection model.In order to complete the pre-task of log parsing,many log parsing algorithms designed according to the log characteristics have been proposed,so as to achieve the purpose of extracting structured logs.Aiming at the shortcomings of the existing log parsing algorithms,this paper improved the system log parsing task allocation model based on distribution and the system log parsing algorithm based on heuristics,and completed the log parsing task by combining the distribution model and the parsing algorithm.Five kinds of data sets,such as HDFS and BGL,are used to verify the feasibility and effectiveness of the model and algorithm,which shortens the time required for log parsing and improves the efficiency of log parsingIn the system log anomaly detection module,a system log anomaly detection model based on GRU is proposed.By extracting log events from logs,the detection model is quickly constructed and the log editing distance loss function is used to adjust the model.This anomaly detection model is used to detect whether the sequence is abnormal,and by comparing with a series of system log anomaly detection methods based on SVM,LSTM and Autoencoder,the validity and feasibility of the system log anomaly detection model based on ED-GRU in this paper is proved.