The Design And Implementation Of Message Queue Operation Platform In Ant Financial

The Message Queue service provides platform independent data communication between distributed systems using efficient,stable,and rich information transmission mechanisms.It integrates distributed systems through forms based on messages.As volume and complexity of the service increase,the user amount and deployment scale of message queue are also growing rapidly:on the one hand,the growth of number of users caused the increase of total time spent on problem support.On the other hand,the increase of deployment size leads to greater pressure on the service governance,such as service status monitoring and service capacity planning.Manual maintenance is encountering difficulties to meet demands as the continuous growing deployment scale reaches a certain threshold.Under the background of the increasing amount of users and expanding scale of deployment of message queue,this thesis did a research on the current situation of products related to message queue maintenance by addressing their weaknesses and combining the concrete requirements,and then proposed a solution from aspects of design and implementation of Message Queue Operation Platform in Ant Financial.The main work of this thesis is as follows:(1)In view of the scalability and maintainability issues of the platform,this thesis proposes a solution of stateless system architecture modeled on functional programming,which extracts the back-end states to a shared and persistent store to solves the complexity problems in cluster expansion,instance migration and instance reboot.This approach realizes the goal to costless add or destruct instances and helps SREs to pay less attention to the platform's correctness,thereby brings great convenience in the platform maintenance.(2)To solve the problem of inadequacy in user supporting caused by the increase of the message queue's user amount,this thesis proposes a solution by integrating the previous user supporting experience into user self-service support service whose main idea is to provide solutions.That solves the low concurrency issue and time consuming problem in user support.More importantly,since the improved support service provides direct guidance instead of pure feedback to help users to solve the problem,thus saves users' effort to analyze the feedback information.(3)For the long kept waiting time problem caused by long establishing time required in SSH connection,this thesis creatively proposes a solution using reusable,reentrant and cache-separable SSH connection pool.Those features above respectively solve the repeated creation of SSH connections,the reacquisition of SSH connections,and SSH connection cache isolation problems.In general,the introduction of SSH connection pool works effectively for both reducing time of business operations and consumption of resources.(4)The growth of message queue in its deployment scale leads to difficulty in service governance.To solve this problem,the thesis proposes a solution about service status monitoring and capacity planning.The status monitoring part helps to get the high dimensional states of the service and to timely detect the abnormalities.While the capacity planning part helps to solve the insufficient cluster capacity and cluster resources wasting problems.Those approaches support the message queue to better satisfy its service level agreement.(5)To deal with the robustness problem in executing tasks in distributed environment,this thesis proposes a distributed task scheduling scheme with lighter weight,which solves the load balancing,tasks missing and task simultaneous repeated execution problems in distributed environment.This improvement helps developers to spare little attention on task scheduling and fault tolerance and focus on the tasks' business logic.The platform has been running in good condition since the system was launched.User self-service support and service governance both work properly.Online statistics indicate that the platform provides service to the enterprise with an average of 9,000 effective page views and 400 users per month,greatly improved the efficiency of message queue in user support and service governance exactly as required.