System Failure Prediction Based On Log Mode Discovery

With the development of information technology,various of complex application systems have been widely used.The operation of the system generates a large number of logs,which record the system status and important events at critical moments,which can help the operation and maintenance personnel to find and solve system failures.However,the existing operation and maintenance system can only diagnose the faults and repair them after they occur,but it cannot avoid the system failure caused by the sudden occurrence of the faults.Therefore,accurately predicting faults can help maintenance personnel take preventive measures to avoid or reduce losses caused by the faults,and it is essential for intelligent operation and maintenance.This article takes log data from application system as the research object,and studies the log pattern analysis method and system fault prediction method based on the heterogeneity of log data,large scale,and unstructured natural language description.A log-oriented fault prediction system is designed and implemented.The contents of research are as follows:1)In order to improve the ability of massive heterogeneous log mining and pattern discovery,a rapid log pattern discovery method is proposed based on the characteristics of log data.The method adopts the One-Pass idea to complete the clustering of all logs by scanning the log data once,and sequentially merge the logs in the cluster to generate a log mode for the cluster.Experiments are conducted on four different scale log data sets and compared with clustering and pattern recognition algorithms in the HLAer framework.The results show that under the premise of ensuring the accuracy of clustering and pattern discovery,the memory overhead is much lower than HLAer,and the running time is about 10% of HLAer.2)Aiming at the problem that the existing fault prediction method requires a large amount of specific system domain knowledge and high-dimensional feature representation,which is difficult to generalize to heterogeneous logs,a fault prediction method based on log pattern discovery is proposed.The method uses the log mode extracted by the log pattern discovery algorithm to re-parse the original log,and then uses TF-IDF to extract the log pattern feature per unit time,and uses the residual connection-based IndRNN fault prediction model for training and testing.Experiments are performed by using two log datasets.Compared with several other algorithms,the experimental results show that the accuracy and recall rate of the model is about 10% higher than SVM and random forest,and about 6% higher than LSTM.3)A fault prediction system based on log pattern discovery is designed and implemented.The system first collects logs of all nodes through the Flume component,then performs fast log pattern discovery on the distributed platform,and trains the fault prediction model based on it.Taking into account the timeliness of fault prediction,the system can obtain logs in real time through Kafka cluster and use the fault prediction model to achieve online fault warning.