Design And Implementation Of Web Log Analysis System Based On ELK

In recent years,the Internet industry has developed rapidly,the scale of the network and the network users have continued to grow.In pursuit of the goal of low cost,high performance,high availability,scalability and high security,the modern web site architecture design uses a lot of open source software,using layered,segmented,distributed clustering,caching,etc.,these complex web site architectures with large number of servers brings great challenges to system operation and maintenance.The massive log generated by various servers is an important information for the operation and maintenance personnel to monitor the system.Log management and analysis are necessary to monitor the state of system,locate the root cause of problem and resolve the performance issue.The traditional log storage method can no longer meet the real-time and flexible monitoring requirements.Therefore,the website operation and maintenance personnel need to build a quasi-real-time log system that can collect logs,perform efficient analysis,visualize logs and quickly locate the problems.In this paper,we design and implement a real-time collection and analysis system for massive logs,combining the characteristics of Internet enterprise architecture and the requirements of log collection and analysis,the system based on three open source software Elasticsearch,Logstash and Kibana(ELK),and we used the logs that generated from hundreds of production servers in an Internet enterprise as the basic data for research.The main work of this paper includes:(1)Based on the mainstream architecture of Web sites,we analyze the characteristics of various types of logs on the Web site and study the analysis methods of the logs.We use Beats to collect logs without intrusiveness,and use Logstash to analyze and process various types of logs,perform field analysis on logs of different formats,and implement flexible log preprocessing.(2)We research key technologies such as real-time log collection,analysis,storage and search of massive logs,we use Elasticsearch to index and store logs,implement a high-performance,high-availability,easy-to-expand realtime log analysis system.(3)By researching the data abnormal point detection method based on statistical model,we combine the log analysis system to scan the system abnormal log regularly,use the information of the configuration management database(CMDB),to detect the abnormal log of the system and provide the statistical information of the error log and speed up the root cause analysis.(4)By making full use of powerful log analysis and indexing functions of elasticsearch,we use Kibana and Grafana to search log content and visualization the log information,and we can use real-time dashboard to combine multiple data sources,to improve company's ability in log management and fault response.