The Research And Implementation Of Real-time Log Management System Under Cloud Platform Environment

In the cloud computing platform,each module distributes around different nodes.Some of them are physical servers and others are virtual machines.There are huge amount of data generated in them.As we all know,log data is of great value.If the cloud platform is likened to a person,the log data is like the various items on the medical examination form after we went to the hospital for medical examination.They give diagnostic information to our cloud platform.However,these data are like the conclusions written by doctors on the medical examination form and are beyond our understanding.Developer often ignore them,let alone tap into their intrinsic value.For this situation,this thesis proposes and implements a log management system that has failure prediction function and can display the result to the user through the web.The system consists of the log collection module,asynchronous communication module,log processing module,storage module and result display module.There are two most core modules on system.One of them is the log collection module based on Flume and the other is log processing module with prediction.The log collection module is the foundation of a full set of systems,which gathers log information scattered on the cloud platform to provide data sources for other modules.For the performance requirements of the log collection module,this thesis has made the following design:(1)The channel has been improved so that it can flexibly select the memory channel or the file channel according to the different data traffic;(2)Hbase_sink is divided into three levels to provide the efficiency of log writing to the Hbase server.(3)A Sink is customized to meet the needs of different modules for Sink.(4)According to the actual operating environment,the parameters are adjusted.When designing the log processing module,this thesis did a series of research on how to mine log information.First of all,log data is different from online shopping data and has its own unique characteristic.Therefore,this thesis has defined a number of new concepts.At the same time,we have studied and proposed an Apriori-like event correlation mining algorithm-Apriori-LTIS.In order to provide system processing efficiency and save resources,and further improve the Apriori-simiLTIS algorithm.Then,an innovative concept ECG(event correlation graph)was proposed to represent the event correlations.Finally,this thesis proposed an ECG-based fault event prediction algorithm to detect possible future failures of the cloud platform.In order to verify the integrity of the system,this thesis performed detailed tests on the performance and functionality of the system,and performed experiments on the proposed algorithm using multiple sets of differentiated data,from the average analysis time,accuracy rate,and recall rate analyze and evaluate the experiment.At the end of the article,the unsatisfactory design of the system is proposed.At the same time,the future optimization direction is elaborated.