Research And Implementation Of Anomaly Detection Technology For Software Systems Based On Massive Log Messages

With the popularization of the distributed system, more and more software applications choose the distributed system consisting of large-scale server clusters as the underling environment, also with the rapid development of cloud computing technology, these software applications gradually choose to deploy in the cloud environment, but the attendant problem is the continuous improvement of the scale and complexity of these software applications,the possibility of system abnormal is increasing. System Log analysis has become the most important means to judge whether the system is abnormal. When the system are failure or abnormal, the system administrator needs to face a lot of system operating information, caused a huge challenge to the detecting abnormal behavior of the system. Log information of the component in the application system record the operation information of the system, so we can tap the massive logs to detect abnormalities of the system. The size and structure of the system log is increasingly complex and large, manually checking has become impractical, therefore there is an urgent need for a tool to automatically find the log messages contain exception information from the massive system logs.Based on the above research problems, in this paper, we combine the use of source code analysis?log preprocessing?clustering algorithm and abnormality detecting methods to design and implement detecting abnormal behavior techniques which based on massive log for the complex software systems. This paper is based on massive log messages and log analysis to improve the existing technical difficulties, the method can be applied to various format system log, solve the problem of finding exception information from massive log messages. The implementation steps of the proposed method in this paper are as follows: The first stage is the preprocessing work before abnormality detecting, the first stage includes four parts: the first part is the analysis of the source code, including traversal the source codes of the system based on using abstract syntax tree and then get the complete set of log templates. The second part is using the Soot tool to traversal all the source codes of the system and then get the reach relationship of log templates. The third part is the preprocessing of the system log, mainly including group the system log by the thread name and then match the grouped log with log templates set. The last part is splitting the matched log in accordance with the group path to get the executive tracks of system log, which based on the reach relationship of log templates. The second stage propose abnormal behavior detection method for the complex software system based on massive logs. The second stage is on the basis of the above preprocessing work and also contains four parts. The first part is get the number of calls between the log templates according to the executive tracks of system log. The second part is using the number of calls between log templates to construct similarity matrix. The third part is to cluster the similarity matrix using improved clustering methods, and the fourth part is extracting exception log template based on the sampling strategy. The last part is get the system log contain exception message from the input system logs based on the exception log template. In the preprocessing stage and detecting abnormal stage we all have carried out experiments, using the Hadoop open source framework verify the correctness and feasibility of the proposed method in this paper. It can be seen from the experimental results that the proposed method for software system anomaly detection based on massive log message has a good effect.