Research On Log Parsing And Correlation Analysis Between Time Series And Event Series

AIOps(Artificial Intelligence for IT Operations)has become a hot trend in the field of operation and maintenance of IT.Root cause analysis in operation and maintenance of IT plays an important role in discovering system faults.Log parsing and correlation analysis in operation and maintenance data have always been the key research contents in root cause analysis.The representative LenMa log parser in the existing research has many advantages such as online-running,no-preprocessing and open-source,but it has a poor performance on dataset HeaelthApp.log;meanwhile,the existing research has solved the correlation detection problem between the burst time series data and the event sequence,but there is no corresponding solution for increasing-trend and periodical time series data that truly exists in operation and maintenance scenarios.In order to solve the above problems,firstly,this thesis analyzes the reasons for the performance of the LenMa parser,and proposes an improved LenMa algorithm.Then,in the analysis of the correlation between security events,two types of correlation detection algorithms between time series and event sequences are proposed respectively.The specific work is as follows:1.The reason for the poor performance of LenMa on dataset HealthAPP.log is found out by the univariate experiments,and an improved algorithm of LenMa is proposed.The improved LenMa algorithm uses the pre-check algorithm to prevent the bad situation of incompatibility when LenMa is running,and uses the log field replacement algorithm to solve the problem of log clustering.After experiments,the accuracy of improved LenMa algorithm on this certain type of data set has been increased from 17.4%to 98%,without decreasing the performance on other data sets.2.A correlation detection algorithm between slow-increasing time series and event sequence is proposed.First,the existence of the slow-increasing time series sequence is demonstrated,and then a correlation detection algorithm is proposed for this new scene of the slow-increase time series sequence.The method of sliding window detection is used for correlation analysis.Finally,the influence of the sliding window size K and the judgment threshold Tr on the algorithm is recorded by experiments.According to the experimental results,the F1 value of the detection effect of the proposed slow-increasing time series and event sequence correlation detection algorithm is 98%,which provides a reliable solution for the problem of slow-increasing correlation detection.3.A correlation detection algorithm between periodic time sequence sequences and event sequences is proposed.Firstly,the reason why the periodic time series cannot be applied to the traditional MTSKNN algorithm is analyzed,and then a correlation detection algorithm is proposed for the new scene of this periodic time series.The algorithm uses the periodic time series detection algorithm for correlation analysis,and calculate the period through discrete Fourier frequency transform,and finally output the most likely period value through anomaly detection.According to the experimental results,the correlation detection algorithm of periodic time series and event sequence proposed can effectively detect the correlation between periodic time series and event sequence compared with the traditional MTSKNN algorithm.The algorithm detected all periodic correlations,F1 value is 100%,and the mean square error of the predicted period value is only 0.011.