Design And Implementation Of Fault Detection System Based On Log Analysis

Log is the record data used by the system to record the running status of hardware,software and problems.It can be used to obtain the running status of the system or check the system problem,therefore its importance cannot be underestimated.In the cloud computing platform,there are many system modules,various types of services,and complicated inter-service invocation relationship,which resulting massive log data.Traditional log analysis method is to analyze system log faults by checking line by line to locate system faults or find code problems.In addition,with the structural diversity and storage diversification of log data,developers are difficult to obtained truly valuable information or quick locate faults from the vast amounts of complex log when screening system malfunction,traditional log analysis methods cannot satisfy the requirement of the fault detection efficiency and productivity.In order to solve this problem,a fault detection system with fast log collection and log analysis is designed and implemented in the framework of cloud platform.The main contents of this thesis include:On the basis of enterprise requirements and research status,combined with the shortcomings of traditional fault detection methods,it is concluded that the system should include the functions of log collection,message subscription,real-time log analysis,distributed storage and visualization.(1)Employing the load-balanced Flume to collect logs and then passing those collected logs from different nodes into Kafka.(2)Adopting Kafka to cache and forward logs,and improving the reliability and efficiency of data consumption by configuring load balancing.(3)Using the Storm framework to realize the operation flow of log analysis and fault detection,implementing fault detection by using the association rule algorithm,which ensures the real-time performance of system data processing.(4)Elasticsearch is used as a storage center and search engine for log data to complete statistical analysis and storage data functions.(5)Based on the MVC architecture,a front-end visual management platform was designed to complete front-end decoupling and visualization.After detecting the performance and functions of the system,it was found that the system was stable and realized the functions of collecting logs,reliable transmission,log analysis,and storing logs.Considering some indicators such as accuracy and credibility,the fault detection algorithm were analyzed and evaluated.In the end,the author of this thesis put forward the shortcomings of this system and the development direction of future work.