A Log Based Algorithm For Network Failure Detection

With the continuous development of science and technology,people have much higher requirements for communication networks.Failure management is an important part of network management.The first step of failure management is to discover the failures in the network.Traditional network failure detection methods rely heavily on alarm data.However,the utilization of alarm data can only analyze the failures that have been mastered within the existing knowledge domain,and it is necessary to construct the knowledge base in advance.At the same time,what is often overlooked is that the network generates a large amount of log data all the time.These log data contain a lot of information.Unfortunately,they are not fully used.This dissertation focuses on logs generated by the network and the usage of logs to analyze network failures.There are two key points when using logs to detect network failures:log representation and failure detection algorithms.Due to the non-structural nature of the logs,log vectorization needs to be performed.Moreover,the representation policy will affect the performance of subsequent failure detection and failure extraction.This dissertation analyzes and explores the log representation and network failure detection algorithms,and designs a complete set of algorithms from log sample representation to adaptive detection of network failures.The main work is as follows:Firstly,a new method was put forward to analyze the current state of the network.Besides the existing alarms and other structural data,non-structured log data,which is similar to natural language,is introduced to determine if the network is normal or not.With samples derived from simulation data,a large amount of prior knowledge and expert knowledge of the network system is circumvented.Thus the algorithm can be widely deployed and the utilization rate of the data is also improved.Secondly,aiming at feature representation problem,this dissertation proposes a multi-granular representation of logs:the word level,event level and event set level.These representation methods do not require fixed format input and manual feature engineering for logs,which overcomes the difficulties of existing log processing method.Based on the representation,in order to analyze the distance between the log texts,the topic word mover's distance(T-WMD)is proposed.Topic information is introduced to the original WMD,which makes the description of the distance between the words more accurate.Representation is necessary to analyze state of the network by logs.Thirdly,based on the representation,the failure self-learning algorithm is designed based on the convolutional neural network.While detecting the known failures,the unknown failures can be discovered simultaneously with the adaptive principle.In the self-learning experiments,the failure knowledge base is obtained by statistically analyzing the output information from the existing samples,through which can we obtain the confidence threshold of each failure so as to determine whether the failure is known or not.The extraction new failure types from the unrecognized samples can also expand the known failure knowledge base,thereby realizing the self-learning process at the same time.The training process can be performed offine in advance while the deployment and operation can be performed online,and the knowledge base can be updated in real time during the operation,so that the log can be processed quickly in the application,and an adaptive detection of the failure can be realized.Experimental results based on simulated data and enhanced data show that the proposed adaptive failure detection algorithm can maintain good performance for both known failures and unknown failures.