Anomaly Detection And Correlation Analysis Based On KPI Data Of A Commercial Bank

With the rapid development of big data technology,the size of network has also increased exponentially compared with the past.Moreover,the efficient and rapid processing of IT operations have gradually become a hot spot.At the same time,due to the popularity of machine learning technology,AIOps(Artifical Intelligence Operations)is formed by combining machine learning methods with operations.In this condition,this paper studies the concept drift detection of KPI(Key Performance Indicator)and correlation analysis based on similarity.The data set is based on the KPI data of a commercial bank.The main work of this paper is as follows:1.Introduce the development process of AIOps,with emphasis on the research status of concept drift detection and relevance analysis for operations data.The significance of this research is expounded.2.A comprehensive method is proposed for concept drift detection.Concept drift refers to the significant change in the distribution of operations data.Due to the fact that the most outlier detection algorithms rely on the collection of historical data in a period of time window for calculation,it cannot adjust after the data distribution changes in time,and false positives usually occur for a pe-riod of time,until the window data completely slides to the changed data and then the false positives can stop.After accurate detection of concept drift,the algorithm can be adjusted in time to efffectively reduce false positives.At present,the relatively perfect method to determine the concept drift is StepWise.In this paper,the concept drift is detected based on this method and the kernel density estimation.Using a commercial bank's 71777 data,the two methods were com-pared.The comprehensive method proposed in this paper effectively improved the accuracy of 29.25%,with a maximum improvement of 17%,and the detection effect of 65.94%data was the same as that of StepWise.3.For the anomaly indicators,the operations staff need to find other indicators with strong correlation with them to investigate the reasons for the occurrence of anomalies.And for some important trading indicators,it is also necessary to analyze whether the changes of these indicators have an impact on other indica-tors.Based on the linear correlation method,this paper calculates the correlation between one data index and other indicators.Different from most correlation anal-ysis algorithms,the correlation between different kpis is invariable.This paper verifies that the correlation between data changes over time.Therefore,it is set to calculate the cross-correlation value by hour/day/week.Furthermore,the phase difference of similar waveform is limited,which makes the calculation speed of the algorithm increase by about 75%.The results of the algorithm give N indexes with the highest correlation.This algorithm has been applied in a commercial bank and its practicability has been verified.Taking the actual data of a commercial bank as examples,this paper improves the existing concept drift detection model and improves the accuracy of concept drift detection.Combined with the actual situation,the relevance analysis al-gorithm is proposed.In the actual operations work,it can effectively help the operations staff to analysis the source of abnormal index,or analyze the possible impact caused by the change of some indicators.