Research And Implementation Of Real-time Log Monitoring And Alarming Platform Based On JStorm

Real-time monitoring of business data has always been a very important requirement for Internet companies.Real-time monitoring can make personalized recommendations,risk management and public opinion monitoring more timely.In the past,limited by the hardware and software costs,this requirement has been difficult to achieve.However,with the development of related theories and technologies for real-time computing,real-time monitoring can already be realized at very low technical cost.Relying on the needs of Internet companies for real-time log processing,this paper implements a JStorm-based real-time log monitoring and alarming platform.This paper describes the development status of real-time computing and the importance of real-time processing of logs for companies.Then introduced related technologies used in system development and highly available system architecture design solutions.Finally introduced the implementation details of the system function module.This system customizes the Source and Channel modules for open source software Flume,making the consumption of Agent resources controllable and the data is not lost after failure recovery.Using Kafka as the middleware for log data transmission,Kafka's related features are used to alleviate the problem of insufficient peak processing capacity of the back-end system.Based on the JStorm development log real-time processing engine,the log processing engine can classify and aggregate the logs according to the rules set by users.Based on the OpenTSDB design data storage format,used to store JStorm aggregated data.The monitoring and alerting service draws on the excellent open source software design methods such as Azkaban and Quartz,and adopts a distributed architecture to ensure high availability of the system.Moreover,the system has developed a front-end management page for displaying,configuring and managing monitoring information.This platform is a one-stop solution for enterprise-level log collection and monitoring.It solves the problems of distributed log collection,real-time processing of massive logs,complex log aggregation rules and real-time alarming.Provides a real-time log monitoring and alarming platform with scalable processing capabilities,guaranteed data security and configurable processing rules.