Research On Anomaly Detection And Correlation Analysis In AIOps

In the era of the Internet and big data,the information revolution in various industries has arrived.The scale and complexity of online services have increased dramatically,which brings huge challenges to operation and maintenance work.How to improve the efficiency and reliability of operations and maintenance,to ensure the effective operation of services in the minimum cost,reducing the loss of enterprises and users due to system failure has become a critical matter.There are two keys in the research of intelligent operation and maintenance : the difficulties in application in anomaly detection and the effectiveness of correlation analysis.The difficulties of anomaly detection are focused on the lack of generality and accuracy,the current research generally ignores the generality of the detection method.In addition,the accuracy of detection needs to be further improved.The effectiveness of correlation analysis is focused on the ambiguity of the correlation object,the real-time,the robustness,and the fragmentation with fault detection.In order to solve the problems,this paper focuses on fault discovery and fault location scenes in online operation and maintenance based on statistics and machine learning to research anomaly detection and association analysis methods,propose effective solutions,with the following main contributions.(1)To address the generality of anomaly detection,the general anomaly detection method OUAD(Online Universal Anomaly Detection)is proposed.By designing a universal fitting feature detector in feature engineering to solve the problem of model selection and adjusting parameters difficulty;by proposing an online weighted sliding window and anomaly measure method,the problem of accuracy due to noise interference and the problem of algorithm generality caused by different scales is solved.The experimental results on real data sets show that the OUAD(Online Universal Anomaly Detection)proposed in this paper has a good performance in accuracy compared to other baseline algorithms.(2)To address the accuracy problem of anomaly detection,we propose the core anomaly detection method OCAD(Online Core Anomaly Detection).The method collect fitting anomalous features and statistical anomalous features in feature engineering and using random foreset improve the accuracy.The experimental results on real data sets proved that the accuracy of OCAD(Online Core Anomaly Detection)is significantly improved the accuracy compared with other baseline algorithms.(3)The associated exceptions detection method OAED(Online Associated Exception Detection)is proposed to address the problem of the effectiveness.The concept of associated exception is proposed to solve the object ambiguity problem of correlation analysis.Using online sliding window and fitting feature detector extract the real-time anomaly sequence ensure the real-time nature.To improve the robustness of the method,research define the anomaly correlation coefficient and calculation to effectively measure the degree and direction of abnormal correlation.The intelligent operation and maintenance application framework IOMF is designed by combining OUAD and OCAD anomaly detection methods to solve the problem of fragmentation.Experiments on real data sets verify the effectiveness of the method.