Research And Implementation Of One-stop Log Monitoring Platform

One-stop log monitoring platform is a one-stop service for log scenarios.Through it,users can quickly complete the collection,consumption,processing,query,analysis,and early warning functions in the log life cycle without development.The one-stop log monitoring platform plays an important role in monitoring alarms,diagnosing abnormal problems,and network security.The current open source monitoring system has made great achievements,but there are still the following problems:(1)Most of the currently deployed data collectors have a single collection capability,and at the same time,the collection task is configured and started with a large workload.(2)Some current log monitoring solutions have certain problems in real-time calculation and processing of massive data.(3)Unable to uniformly manage the complete cycle of the log monitoring system.This article builds a one-stop log monitoring platform for the above problems.The main research contents include the following three parts:1)Research and implementation of BeatWatcher-based multi-dimensional data collection management system,mainly including self-developed Go-based Beat Watcher,to simplify the configuration and startup of workloads in data collection tasks while achieving log data Multi-dimensional collection capability;2)Research and implementation of Flink-based log data processing system,using Flink as the log data processing engine,and real-time processing of massive log data by real-time consumption of data from the distributed message queue Kafka;3)based on The research and implementation of Elasticsearch's visual monitoring and analysis system mainly includes the realization of log warehouse,log retrieval,multi-dimensional analysis,and the ability to monitor and alarm.At the same time,by combining the above.Realize unified management of the entire monitoring system from machine management,data collection,data distribution,data processing,storage,retrieval,analysis to data alarm.This article designs and implements a one-stop log monitoring and analysis platform,completes data visualization and system testing,sets up a unified visual web interface,provides unified visual management of the entire one-stop log monitoring platform,and all log data Multidimensional visualization and analysis capabilities.At the same time,the system was built in a distributed cluster and various functional modules were tested.By analyzing the system test results,the system's functional integrity and performance indicators were verified,which proved that the system has the characteristics of availability,reliability,and easy expansion.This system has been deployed and used in the laboratory's big data cluster.