Research On High Precision Log Parsing Techniques

Program logs keep track of key events and warnings during the run-timeof a program. Although they contain valuable information for programdiagnosing, in large-scale systems, logs are rarely utilized in debugging due tothe huge number of modules and the difficulty in understanding logs.Researchers have made quite a number of attempts to deal with thisproblem. They combine source code and program logs to dig usefulinformation out of the log to help debugging. Among these works the mostimportant technique is log parsing. It transforms log text to structuredinformation and establishes its connection to source code. Log parsing isessential to any log-related work in that it provides data to the subsequentanalyses. However, so far there is no common way of implementing logparsing. In existing works, log parsing is done in a fairly intuitive way, whichresults in low precision. Since log parsing feeds data to its subsequentanalyses, the low precision in log parsing will have a bad effect on the resultof the whole work.In this paper, we attempt to deliver a log parsing technique with highprecision. We did a lot of study on the state of the art log parsing techniques and proposed a novel methodology based on static analysis. We perform flowanalysis on source code and construct a finite state automaton for each logpoint. These automata describe the language of a log point in a fine way andare later used to match log text. A comparison experiment with existingtechniques shows that the new approach significantly improves the precisionof log parsing.The main research content and achievements of this paper are as follows:(1) In this paper, we did deep research on current log parsing techniquesand propose a framework of log parsing. The framework consists of3modules, which are log point detection, log pattern construction and logmatching.(2) We also proposed a novel log pattern construction algorithm, which isbased on program control-and data-flow analysis. It first transforms theprogram s class files into an intermediate program. Then the intermediateprogram is transformed into a multi-level automaton through a series oftransformation. A definite automaton is extracted for each log point from themulti-level automaton to represent the language of that log point. The newmethodology combines intra-and inter-procedure program analysis and isable to generate fine level automata.(3) We designed a comparison experiment against current techniques andverified the performance of the new log parsing technique. The result of the experiment shows that the new technique yields a20%higher match rate thanthe old one with acceptable extra time overhead.