Research And Development Of Key Technologies Of Cloud Service-oriented Log Processing System

The current software system is developing towards cloud-native and micro-services,which makes the cloud service module numerous and the interaction between the modules complicated,and the reliability guarantee of the cloud service becomes very difficult.Generally,system reliability guarantees mainly rely on system runtime logs.These log data are scattered on different machines and record the status of the system operation.They are important data sources for developer and operation engineer to debug the system and diagnose failures.Therefore,the log data needs to be collected and processed reasonably.In this case,a logging system was created.The current log system generally consists of log collection,log processing,log storage,and log query modules.These modules cooperate to effectively collect and store logs and solve the problem of centralized storage of logs.Despite this,the current log system still faces many problems.The manual processing method results in log analysis is inefficient and error-prone,and it is difficult to cope with the increasing complexity and scale of the current cloud service system.The log data itself will not produce value,and the processing and analysis of the data will produce value.Therefore,how to improve the automation and intelligence of log data processing on the existing log system architecture is an important part of the reliability guarantee of the cloud service system,and it is also a place where the current log system architecture can improve and optimize.This is main starting point and innovation of this thesis.The cloud service-oriented log processing system proposed in this thesis focuses on the log data processing stage in the traditional log system architecture.The main work is divided into two parts:(1)research on the key technologies of real-time log analysis and real-time anomaly detection;(2)design and implement a logging system based on the research.The difficulty of log analysis is that the log format is not standardized,and it is difficult to describe it with uniform rules.The Fog method proposed in this thesis matches the logs by constructing a parse tree.Users only need to define a few rules to match most formats.The difficulty of anomaly detection is that the log data is multi-sourced and non-standard.The previous methods focused on execution path anomaly detection,and mainly offline algorithms.The Intelli Log proposed in this thesis uses deep learning technology for anomaly detection,which not only covers execution path anomalies,but also takes into account the system status value,and it is an online algorithm.The difficulty in designing a log system is the large amount of data and high real-time requirements.This thesis divides the log system into two subsystems: offline log processing and real-time log processing.With the help of stream computing and other technologies,it can handle large amounts of data and achieve real-time requirement.By analyzing and processing the log data of the cloud contact center,the following results are obtained in this thesis.First of all,the log analysis method Fog proposed in this thesis can deal with free-form logs,and the performance reaches a processing speed of 10,000 per second under a single thread.In addition,it supports parallel acceleration.Secondly,the Intelli Log anomaly detection model has a good effect compared with the previous method.In terms of performance,it can process more than 12,000 data per second under the CPU.An analysis of the anomaly case is also given.Finally,the actual log analysis and anomaly detection delay of the log system designed and implemented in this thesis is between 1.5 and 5 seconds,and the real-world experience of the system on the cloud contact center is also given.These results show that the system proposed in this thesis can meet the log processing requirements of cloud services,and has novelty and practical value.