Research On Anomaly Detection Of Cloud Data Center Server KPI Based On Machine Learning

The cloud data center has a huge system architecture and a complex and diverse user group.Operators need to detect anomalies in KPI data to ensure the reliability and stability of systems and business in the cloud environment.However,due to the large number of KPI data monitored by the cloud data center,the process of marking the KPI,selecting the anomaly detection model,model training and parameter tuning will consume a lot of labor costs.To solve this problem,this thesis builds an intelligent KPI anomaly detection strategy based on clustering algorithm and unsupervised anomaly detection algorithm.(1)Since the KPIs of the same monitoring index are highly correlated,after clustering the KPIs of the same index,the centroid KPI of the cluster is similar to the normal patterns of other KPIs in the cluster.At this time,only the centroid KPIs need to be trained and parameters tuned.KPIs in the same cluster can share the anomaly detection model,thus reducing the workload of anomaly detection.Therefore,this thesis proposes a KPI clustering algorithm based on AKRNN-DBSCAN.This algorithm uses the reverse k-nearest-neighbor of KPI data to determine the neighborhood density.It does not need to input parameters manually,automatically find the k-value interval with stable number of clusters,and sets the minimum k-value when the number of clusters reaches stability for the first time as the optimal solution,at this time the proportion of unrecognized KPIs is the smallest.The results show that the AKRNN-DBSCAN algorithm can effectively cluster five monitoring indicators such as CPU utilization,the F-score is between 0.78 and 0.90,most of the KPIs are recognized,and the proportion of unrecognized KPIs is only between 5% and 17%.At the same time,the clustering effect is better than the traditional K-means and DBSCAN algorithms.(2)For the problem of supervised learning anomaly detection algorithm requires labor and time to mark KPI data,this thesis proposes an unsupervised anomaly detection algorithm of LSTM-Autoencoder based on time series feature extraction.It enriches KPI time series features from three aspects: statistical,morphological and entropy characteristics,captures the changes of different dimensions in KPI t ime series,and enrich the feature space of the original KPI time series;LSTM is used to capture the dependency between KPI time series;Autoencoder reduces the dimension of the input data,obtains the most representative feature of the input data by using small feature space,reconstructs the input data according to the representative feature,and judges the anomaly by the error between the original input data and the reconstructed data.The experimental results show that the LSTM-Autoencoder after time series feature extraction has achieved good results in anomaly detection on five monitoring indicators such as CPU utilization,the F-score is between 0.93 and 0.97,and the algorithm is superior to the anomaly detection algorithm using the original time series.(3)For anomaly detection of large-scale KPI data,in order to improve operation efficiency and reduce the cost of anomaly detection,this thesis combines the above two algorithms,first finds the centroid KPI of each cluster after AKRNN-DBSCAN clustering,and then uses LSTM-Autoencoder anomaly detection algorithm based on time series feature extraction to train the centroid KPI anomaly detection model,which is shared by other KPIs in the cluster.The experimental results show that,compared with the previous method of performing abnormal detection on all KPIs,the five monitoring indicators such as CPU utilization are shortened in time by 84%?93%,and the F-score is only reduced by 13%?16%.