The Design And Implementation Of Microservice Anomaly Detection System Based On Ensemble Learning

With the development of microservice and the wide application of machine learning in various fields,intelligent microservice indicators anomaly detection has become a new research hotspot in industry.Traditional anomaly detection rules based on static thresholds are costly to manually configure,and when business scenarios change,the thresholds must also be changed accordingly,which can no longer meet the needs of microservice anomaly detection(MSAD)in complex scenarios.Most current anomaly detection algorithms are unsupervised learning,and their detection effect is not good.If supervised learning is used,although the anomaly detection effect is better,the cost of label data is too high.In this thesis,in order to reduce the cost of configuring MSAD rules,combined with actual needs,a MSAD system based on ensemble learning is designed and implemented.The main work of this thesis is as follows:(1)A low-coupling MSAD system architecture is designed.In the technical implementation,the design pattern of the microservice architecture is adopted to achieve frontend and backend separation,backend componentization,and multi-instance deployment,ensuring the scalability and availability of the system.The job scheme is used to divide the backend into three components: API Server,Scheduler,and Processor,which are responsible for job management,job scheduling,and training detection,which reduces the coupling between components.(2)An anomaly detection method combining unsupervised ensemble learning and supervised ensemble learning is proposed.Unsupervised ensemble learning integrated with i Forest,OCSVM and LOF is used to generate cold start labels for the raw time series data.After the cold start,the random forest algorithm is used to obtain an anomaly detection model by training the labeled time series data.The experiment results show that the quality of the labels generated by unsupervised learning after integration optimization is significantly improved compared to before integration,and as users continue to feedback on anomaly detection results,the model detection effect will become better and better.(3)A practical MSAD system is implemented.It is divided into two modules according to function: training detection module and job management and scheduling module.The functions of the training detection module include time series data preprocessing,generation of cold start labels,training detection models,and executing anomaly detection,which are implemented by the Processor.The functions of the job management and scheduling module include job management,job scheduling,and job feedback,which are implemented by the API Server and Scheduler.Implement concise frontend,providing an entry for job management and feedback.The functional requirements and non-functional requirements of the system are tested,and the anomaly detection results are analyzed to verify the effectiveness and practicability of this system.