Algorithm Research And Implementation On Real Time Anomaly Detection Of KPI Data

KPI data(Key Performance Indicator)is time series data obtained by timed sampling with practical application significance.Monitoring KPI is important for enterprise applications: by monitoring KPI data in real time,user can find anomaly in time and do action to ensure the normal operation of the application.In enterprises,the method of real-time anomaly detection by setting thresholds for KPI data is very common.However,the threshold setting depends on the user's experience.At the same time,as the KPI data increases,the method of configuring several thresholds for each KPI data is no longer used.In order to achieve anomaly detection for KPI data and achieve the goal of threshold-free setting and high automation,this topic has carried out related work and proposed three algorithms.These three algorithms are highly automated and have high use value and practical significance,so that users do not need to set thresholds for each KPI separately.The main work of this paper includes:Firstly,an anomaly detection algorithm based on supervised learning and anomaly threshold separability is proposed.We give the concept of "anomaly threshold separability",using this concept,we propose an anomaly detection framework based on supervised learning.The algorithm automatically extracts and learns the characteristics of each KPI data.After comparison with mature algorithms and systems,our method can increase the F1 score by more than 13%.Secondly,an anomaly detection algorithm based on extreme value theory is proposed.In order to realize the threshold-free anomaly detection,this chapter applies the extreme value theory.It does not need to make any assumptions about the distribution of the original KPI data.It accelerates the computational efficiency by introducing moment estimation,introduces other prediction algorithms to improve the algorithm effect,and introduces quantitative update to deal with the concept drift problem.Experiments confirm the effectiveness of our alg orithm.Finally,an unsupervised anomaly detection algorithm based on dynamic voting is proposed.Combined with the characteristics of KPI data,we give the concept of "local area" for KPI data,making the base detector which experts at specific data has higher weight.Then dynamic hard voting algorithm and dynamic soft voting algorithm are given respectively.The experimental results show that we can increase the F1 score of the original system by more than 4.