Design And Implementation Of Prism Monitoring System

With the continuous expansion of department business,as well as the transformation of product architecture into service architecture,the monitoring of service performance becomes more and more difficult.Tranditional performance monitoring schemes have been unable to meet the demand for service monitoring,because their disadvantages become more and more obvious.Such as the monitoring is not centralized,the standard is not uniform,the dimension is not comprehensive,and strong rely on business.This paper aims to set up a common,non-business-coupled service performance monitoring and analysis platform that provides a unified performance monitoring solution for Java services of each product in the department.In the research and development processes of the prism monitoring system,the author participated in collecting the system requirement,forming uniform monitoring standards and dimension,and designing the system architecture and database.After that,the author divided system into 4 subsystems:data acquisition,data collection,background management,monitoring and alarm.The data acquisition subsystem is used for the acquisition of performance indicators,including service response time,method execution time,sql execution time and so on.The data collection subsystem is a web service,which responsible for collecting the performance data acquired by data acquisition subsystem and adding data to elasticsearch.The background management subsystem including product management,service management,service report,log query,alert rules management,alert records management and some other functions.The monitoring and alarm subsystem is responsible for monitoring the stability of the system and the periodic execution of alert rules.The author has independently completed the design,implementation and unit testing of data acquisition subsystem,data collection subsystem,monitoring and alarm subsystem,and the back-end part of the background management subsystem?In order to realize zero coupling between the monitoring code and the business code,this paper designs a performance data acquisition scheme based on the technology of modifying bytecode dynamically.The scheme uses the aspect-oriented programming idea,through framework adaptation,configuration file and other ways to define the entry point,and can be accessed by setting-javaagent parameters when the service startup.After that,the monitoring code will be weaved into the class at the entry point before a class loaded,so that we can add performance monitoring function without modifying any business code.The scheme can collect the response time,method call stack,sql execution time,request parameters and other data,and also supports the monitoring of rpc call and asynchronous call.This system has been launched in Jan 2017,still stable operation so far.It has provided performance monitoring service for 71 services of multiple departments in the company.The system can accept more than 80 million,nearly 300G log data every day,and has helped to location system bottlenecks and troubleshoot online problems for many times.In a word,the results of this project far exceeded expectations.