Anomaly Detection Technique And Monitoring System Based On Log Monitoring

With the continuous advance of computer science and internet technology,the information industry has evolved into a new stage.Both the scale and complexity of software and hardware that support the operation of our society,have reached a level never seen before.Governments are now carrying forward the plan of "Internet+",which consolidates the link between the Internet and traditional industry and accelerates the evolution of each other.In the era of cloud computing and big data,it becomes more and more important to maintain the runtime stability and detect potential anomalies in the large-scale software & hardware systems.Log monitoring is one of the most important part in software monitoring systems.People could judge the runtime status of the system and predict possible anomalies by the analysis of log contents and the feature of log outputs.There have been lots of researches on log analysis,existing log monitoring systems have also obtained good results.But current monitoring techniques are still facing some problems.1)The volume of data generated from log files can be really large.A large-scale software system could generate terabytes of logs per day.To analyze the log contents line by line would put big pressure on the system.2)Log messages are unstructured and may be incomplete.The payload of log data does not follow a specific format,and logs may get lost when the system is under a heavy load or error,which makes it inconvenient for machine process.3)Logs generated from different systems differ a lot in the format and payload,a general monitoring solution may not achieve a high accuracy in detecting anomalies.In order to solve these problems,we did lots of researches and experiments,then put forward a behavioral anomaly detection approach based on log monitoring,and further implemented a general log monitoring system.Our work in this paper involved three contributions:1.A log preprocessing method based on log normalization and message clustering.We first normalize the log data,then categorize logs into clusters by the similarity of the payload,with an effective hierarchical clustering algorithm.2.A general log anomaly detection method based on behavioral anomalies.We extract specific behavioral patterns from clustered logs,detect anomalies in log sequences based on the anomaly score and similarity of those extracted patterns,so as to predict the system status.3.A log monitoring system based on the technique of behavioral anomaly detection.The behavior model via training analyzes log streams in an instant and efficient way,which helps the administrator monitor and maintain the production environment.