A Study On Log-based Server Failure Analysis

For the ever-increasing scale and complexity of servers,it is hard for developers to conduct manual analysis to trace faults,which is also probably not practicable to diagnose the causes of faults.So,it is indispensable to develop automated fault analysis tools.Automated fault analysis tools need to rely on a type of data,which records runtime information of servers.To satisfy the need,we utilize logs,which contain plenty of rich information such as the timestamp of an event and the state of a task execution.As logs are usually textual for convenience and flexibility,processing natural language must be developed.To meet the above needs,an automated log analysis tool,which is able to process natural language,is needed.To address this issue,in this paper,we propose a framework for log analysis.The contributions of our work are here: 1.A framework for automated log analysis is proposed,which consists of log parsing module,word to vector conversion module,model training module and log analysis module.The key function of the framework is to preprocess raw logs and output analysis results generated by the deep learning model.This framework is applicable to automated log analysis for a single system.2.Dynamic memory network model achieves good performance on question answering tasks.As a result,we modified the DMN model and built an improved dynamic memory network,which includes rule-based inference module.For better performance on log analysis,the improved dynamic memory network combines expertise,knowledge and natural language processing.3.Due to the fact that logs are usually enterprise data and few logs are shared,we generate artificial logs based on a few opensourced logs and bAbI tasks introduced by Facebook researchers.4.We conduct experiments with dynamic memory network model and improved model proposed in this paper on artificial logs.Experiments show that the improved model takes 1-3 epochs less than that taken by original model to make the loss decrease and be close to zero.To put it another way,the improved model achieves a faster convergence than original model.Another key thing to remember is that,when training the model on 3-9 epochs,the accuracy of the improved model is about 3%higher than the original model.In other words,the improved model is more accurate.