KPI Anomaly Detection Based On Multi-dimensional Feature Extraction And XGBoost

Key performance indicators(KPIs)come from system logs,web page visits,or other data sources.Analyzing the KPI data to track when the system is abnormal,and using the regularity of its time series to predict the system exceptions,can help the operation and maintenance personnel to discover the unknown risk and reduce loss caused by system exceptions.However,due to the lack of abnormal data in KPI and the diversity of fault types,KPI will fluctuate periodically and stably with time,which poses a great challenge to design an efficient anomaly detection model with strong generalization ability.This paper studies the single dimension KPI anomaly detection method from the perspective of multi-dimensional feature extraction and anomaly detection model construction,aiming to find an effective feature extraction method and anomaly detection algorithm to improve the accuracy and generalization ability of anomaly detection model,thus further strengthen the ability of operation and maintenance personnel to deal with program or system faults.Based on the analysis and research of KPI data characteristics,wavelet analysis,XGBoost,other theories and technologies,this paper makes an in-depth study of KPI anomaly detection technology lying on multi-dimensional feature extraction and XGBoost.The main work completed is as follows:Firstly,a KPI anomaly detection framework based on multi-dimensional feature extraction and XGBoost is designed.The framework gives the entire process of KPI anomaly detection,with a purpose to improve the accuracy of anomaly detection of KPI anomaly detection models on various types of single-dimensional KPI data;Then,the statistical characteristics,fitting characteristics and original characteristics of single dimension KPI time series are defined.Statistical features include concavity and convexity,first-order difference and occupancy ratio.Fitting features include peak coefficient,coefficient of variation,moving average,differential moving average and exponentially weighted moving average.These features can well reflect the degree of data dispersion,change trend,correlation and implicit characteristics;After that,multi-dimensional feature extraction method based on principal component analysis and wavelet is given and experimentally verified to be effective compared with other related methods.In this method,principal component analysis is used to reduce the dimension of the first extracted feature to avoid linear interference between features.At the same time,wavelet is used to decompose the dimension-reduced data according to the time sequence pattern to further extract the non-stationary features of the data;Next,an improved XGBoost-based KPI anomaly detection algorithm is proposed.In this algorithm,smote oversampling and random undersampling techniques are used to equalize the extracted feature set,which is conducive to the learning of sample data;a KPI anomaly detection model based on XGBoost is constructed,and artificial fish-swarm algorithm is used to optimize the learning rate,the maximum depth of decision tree,the minimum leaf node sample weight and regularization weight of XGBoost to ensure that The optimized parameter combination is the global optimal combination,which further improves the anomaly detection effect of XGBoost;In the end,by selecting open datasets,this paper compares the algorithm of KPI anomaly detection based on multi-dimensional feature extraction and XGBoost with some other related methods.The experimental results show that this method can accurately identify the anomaly in a variety of types of KPI sequences with high recall,precision,AUC scores,and good generalization ability.