Research And Implementation Of Micro-Service System Monitoring And Failure Prediction Based On Log

The Internet architecture has undergone several changes from a single-host,layered architecture to a later cluster architecture.The architecture has always changed with factors such as changing business application requirements,and microservices emerged during this change.The micro-service architecture breaks through the traditional virtual machine-based cluster architecture model.According to certain strategies,the traditional single application is split into multiple micro-services.The container is used as the carrier,and the containers interact with each other through a lightweight communication mechanism.It solves the ever-increasing system performance requirements of applications and improves R&D efficiency.However,the micro-service granularity in the actual application scenario is smaller than that of the traditional virtual machine,and the scheduling is more complicated.At the same time,with the development of the service,the scale of the micro-service is continuously expanded,and the relationship between the micro-services is more complicated,so that it is whether for the maintenance personnel or application developers,as a result that the difficulty and cost of monitoring and managing the system are greatly increased.Based on the above characteristics and background,this thesis studies the key technical issues of log-based microservice monitoring and fault prediction.This thesis designs and implements a log-based microservice monitoring and fault prediction system,which is mainly composed of microservice log collection,microservice monitoring and microservice fault prediction.Firstly,based on Fluentd,the high-availability and low-latency log collection function is implemented,and the collected log data is taken as input to analyze and process to realize micro-service monitoring and fault prediction.The monitoring of microservices is mainly divided into three aspects:(1)microservice overview and performance indicator information.Used to obtain micro-service running profile status information,such as micro-service API(Application Programming Interface)feature set,micro-service processing time-consuming and other information.(2)Microservice call link tracking.The real-time analysis and calculation of the micro-service log is performed by using the Storm,and the link through which the user is called is drawn according to the calculation result set.The link contains information such as the complete path of each container experienced by the request,the time required to request the network,the processing time,and the abnormality of the upper and lower levels.(3)Microservice fault tracking and location.Through the weighted-based frequent item association analysisalgorithm proposed in this thesis,the logs in a specific time range are analyzed to effectively help users locate and troubleshoot faults.Microservice failure prediction.The log data is divided based on the time interval unit,the features are extracted and the features are filtered,and then the filtered result set is input into the classification prediction algorithm to train and learn,thereby predicting the fault of the micro service.Through this method,the fault of a certain period of time in the future is predicted.Finally,the system is tested from the aspects of micro-service monitoring and micro-service failure prediction.The test results show that the log-based micro-service monitoring and forecasting system implemented in this thesis meets the design requirements and can achieve low latency and host resource occupancy.Low-level,high-performance monitoring requirements;Withing the analysis of log data,the micro-service monitoring and system fault prediction can providing valuable information for operation and maintenance staff to minimize business failure losses.