Research On Anomaly Detection And Analysis Based On System Log

System log data plays a key role for system administrators to manage the system and diagnose system problems.With the explosive growth of data,manual analysis of system logs not only consumes human resources,but its analysis speed has lagged far behind the speed of log generation.The use of automatic detection technology based on machine learning methods has become the current development trend to solve the problem of abnormal detection of system logs.However,the effect of traditional machine learning methods in the face of changing data and overlapping data is not stable.As the system continues to run,the logs reflect the current state of the system and generate new knowledge.In order to adapt to the ever-changing log data,this paper proposes a system log anomaly detection model based on conformal prediction algorithm.Combining statistical learning with machine learning methods,the framework of conformal prediction is introduced into the problem of system log anomaly detection.Compared with the detection method based on the static threshold,the statistical learning method of the conformity measure can dynamically adapt to the changing log data.In addition,the computational efficiency of the statistical learning method for conformity measurement was improved.As a result,the system log anomaly detection model based on the conformal prediction algorithm can quickly and accurately detect the anomaly of the system log.Because the attribute values of the system log data are too similar,it is impossible to accurately divide the decision boundary,and the phenomenon of class overlap occurs.Aiming at the log data with the class overlap phenomenon,this paper proposes a class overlap anomaly detection model based on ensemble learning.First,the relationship between the sample data and the membership of different classes was calculated,and the data of the overlapping regions were extracted by using the fuzziness to reduce the impact of the non-class overlapping data on the experiment.AdaBoost,an ensemble learning approach,is used to detect overlapping data.Compared with traditional machine learning algorithms,AdaBoost can better classify the log data of the overlapping areas through the iterative basic algorithm,so as to achieve the purpose of detecting the anomaly of the system log.