Research And Implementation Of A Real Time Log System In Cloud Environment

In the current environment,cloud computing has become an important service delivery model by relying on the advantages of effective data flow.However,due to the continuous increase in the scale of system integration,the number of node components,and the number of applications,the number of log records generated by it is also increasing.With a dramatic increase,it is necessary to design and implement a log analysis system in a cloud environment to manage log data.In the log system,the same or similar log records may originate from different nodes and different processes,resulting in high data redundancy.This imposes a huge burden on data analysis.Therefore,filtering log events is an indispensable processing step for data preprocessing in the system.In addition,the system log is usually the first source to obtain internal system operating conditions and system fault diagnosis.Currently,in the cloud environment,most of the fault prediction technologies are based on system operating parameters or fault symptoms,and the fault instance is singled.The predictive analysis of dimensions cannot meet the real-time and effectiveness of fault prediction for the operating status of the log system in the cloud environment.To solve the problem of high redundancy of system log data and low accuracy of fault prediction in cloud environment,this paper proposes the following research:(1)For the problem of high system log redundancy in the cloud environment,this paper proposes a system log preprocessing method based on dual filtering mechanism.In this mechanism,the filtering technology of time filtering window and space similarity pair is adopted,and the system logs that are related in time and space are filtered to reduce the redundancy of the log data.Through experimental tests,when the time thresholds of the temporal window of filtering and spatial similarity take 60 ms and 0.5 respectively,system failure prediction and analysis are performed on the system log after the dual filtering mechanism and the original system log,and the system's failure prediction accuracy rate has no effect.This filtering mechanism is effective.(2)In the real-time log system based on dual filter mechanism,the problem of predicting the accuracy of system faults caused by single-dimensional analysis of event instances is low.This paper proposes a method of online fault prediction under the cloud environment.(CE-OFPM).The method performs multi-dimension real-time calculation and analysis of fault instances.It uses the fault log event as the sample data set to establish the HMM adaptive expansion modeling,and uses the Baum Welch expectation maximization algorithm to optimize the model parameters and adopts improved Bayesian classification algorithm for system state prediction.In order to conduct comparative analysis,the accuracy,recall rate,F-measure,ROC curve and delay time were used to analyze and compare the three typical failure prediction techniques.The accuracy of the system failure prediction achieved through this experimental scheme was analyzed.The highest can reach0.89.The experimental results show that our CE-OFPM method is very effective in predicting faults online.