Research And Application Of Log-based Event Mining Methods

In recent years,various IT systems have become increasingly large scale and more complex and generate massive logs in production environment.Logs record system status information during running.The analysis of logs can help system administrators to know the operation status and also help the administrators to diagnose the cause of fault and error,optimize the system performance and alarm abnormal abnormalities.It is a very difficult job to analyze logs only by man,so we can use log mining technology to help administrators.The main research contents are as follows:(1)By analyzing the form of log data,we design an improved method based on log signatures to generate events.It can convert unstructured log data into structured log data?There are no special requirements for log text format,and the number of event types can be determined autonomously by using log signatures to identify event types.(2)We study the method of mining lag intervals in sequential pattern and the suitable lag interval can improve the accuracy of sequential pattern.To avoid scanning the event sequence repetitively,we transform the event sequence into the sorted table and generate the sorted table by double loop and binary search to reduce the time complexity,so we can get a better time complexity.(3)In order to describe log data in both global and local perspectives,the event summary description model based on the frequency of event occurrence presents the idea of global model and local model.Four methods of generating the description model are designed.Compare and analyze the complexity and compression ratio of these four methods to choose the optimal solution.(4)Based on the above research,we design a log event mining platform to mining and analyze log data.This platform provides log management,log statistics,pattern mining and event summary,and visualizes the analysis results.