Research On KPI Anomaly Detection Techniques Based On Machine Learning

With the continuous growth of Internet companies' business and service scales,it has become very difficult for traditional network management tools to effectively manage and maintain the network to ensure the security and reliable operation of network services.In network management and maintenance,KPI(Key Performance Indicator)anomaly detection is an important basis for network managers to perform further management decisions.In general,network managers need to select appropriate anomaly detection algorithms,and analyze the detection results replaced by step-by-step algorithms to discover potential problems in the network in a timely manner.In order to solve the challenges and difficulties faced by KPI anomaly detection tasks in real-world network maintenance,this research is based on the particularity of KPIs.For the KPI anomaly detection,the data is generalized with feature embedding and feature augmentation.Problems such as feature dimensionality reduction and model migration technology have been studied.This thesis mainly completed the following work and contributions:1.A generalized feature extraction method and unsupervised feature augmentation technology for KPI are proposed.This paper proposes a generalized feature extraction method for KPIs based on the analysis and analysis of KPI samples in real scenarios.And proposed three methods for feature augmentation of original feature data using three fast unsupervised anomaly detection algorithms.Improved the generalization performance of KPI feature representation in multiple anomaly detection algorithms.Experiments show that the generalized feature extraction method and unsupervised feature augmentation technology proposed in this paper can achieve better results in KPI anomaly detection tasks.2.A feature dimensionality reduction technique based on the interpretation of feature contributions of integrated models is proposed.Based on the explanatory research on the integrated anomaly detection model,this paper uses SHAP values to analyze different features of the data,and proposes a feature dimension reduction technology based on model interpretation.Experiments show that the proposed method can significantly improve the training speed of the XGBoost anomaly detection model while ensuring the accuracy of the anomaly detection results.Being able to obtain an intuitive explanation of the anomaly detection results helps network managers to further analyze and summarize the anomaly detection problems in network management in combination with the actual business background.3.A fast time series clustering algorithm for KPI is proposed.The corresponding model migration strategy is designed based on the similarity analysis of KPI curves.The KPI clustering algorithm is used to analyze the migration of corresponding models between different instances.Based on the results of clustering similarity analysis to guide model migration,it can better solve the problem of scarce anomalous samples and difficult labeling in actual scenes.Significantly improves the utilization of labeled samples during model training,Experiments show that this study proposes that transfer learning technology can effectively improve the applicable scenarios of supervised anomaly detection models.And reduce the overhead of model training and data labeling.