Design And Implementation Of Monitoring And Management System For Kafka Message Middleware

In distributed systems,message middleware is a very important component,which can achieve application decoupling,asynchronously processing,log processing and so on.As a traditional distributed message middleware,Apache Kafka provides higher performance than other message middleware by means of publishing or subscribing.And it is very popular with most companies,which can be used in many fields.However,how to better monitor and manage Kafka clusters and how to improve the convenience of using Kafka are all real problems faced by companies.Therefore,in order to provide an enterprise solution for the management of Kafka,a comprehensive monitoring management system is needed.It is not only convenient for operations to maintain the cluster,realizing the monitoring and management of clusters metadata,but also can help ordinary users to use Kafka.The monitoring and management system for Kafka mainly contains Kafka metadata collection module,cluster management module and cluster monitoring module.Firstly,the Kafka metadata collection module contains the collection and real-time updating of cluster metadatas such as topic(a message queue),broker(a server),consumer,controller and metrics.The topic,broker,and consumer information are mainly obtained through ZooKeeper and Broker.Metrics information is collected by JMX technology.Cluster management module and cluster monitoring module are all based on metadata collection module.Secondly,the cluster management module is mainly used by operations,and can realize cluster management,cluster metadata management,machine management,and topic order management.Finally,the cluster monitoring module is mainly used by ordinary users.Users can check topic information and consumer information,set up alarm configuration,and submit creation expansion topic or consumer group order.The cluster management module and the cluster monitoring module adopt the B/S architecture.And users can submit a corresponding request in the web interface,then background returns the response.The background's architecture is based on SSM.And the system's relationship data is stored by MySQL database.The system's development is based on the flow of requirements analysis,systemdesign,system implementation,and system testing.Each stage is carried out in strict way.In the end,the system achieved good results,realizing the monitoring and metadata management of Kafka clusters,reducing the threshold that ordinary users use Kafka and improving the efficiency of operations.