Research On Load Balancing Technology Of Distributed Message System For Cloud Services

With the vigorous development of the Internet and big data technologies,the era of cloud services has arrived.Cloud message service is based on distributed message system,which realizes the transmission of messages between distributed systems and solves the problem of accessing massive real-time data.However,when large-scale data access to distributed message systems under cloud services,the problem of load tilt gradually emerged.The imbalance in load among service nodes affects the availability of cloud messaging services,making it more difficult for service providers to meet quality of service agreements.Aiming at the common load tilt problem of multiple mainstream distributed messaging systems including Kafka,RocketMQ,and RabbitMQ,this article mainly completed the following tasks:(1)Aiming at the characteristics of the distributed message system,the capacity target and the distributed balance target are taken as the two specific goals to realize load balancing.The CPU utilization,disk utilization,network inbound rate,and network outbound rate are the main indicators of load monitoring.Based on the role types of different loads,the leader role migration strategy and replica migration strategy are designed.For the two load migration strategies under the two goals,the principles and algorithm flow for selecting the target replica and the destination node are designed in detail,so as to ensure the consistency of data and the availability of partitions under the distributed load migration.Finally,the general algorithm flow is explained,and the parameter impact and cost evaluation are discussed.(2)The design and implementation of the load balancing strategy is based on the premise that the utilization of nodes and replica resources is known.Aiming at the problem that the existing distributed message system cannot directly obtain the CPU load of the replica level,a load-aware algorithm based on linear regression is proposed.Based on machine learning algorithms,the implicit correlation between the CPU utilization at the service node level and the network inbound and outbound rates is modeled and analyzed.The learned linear model is used for the leader copy and follower copy,respectively,to obtain replica-level CPU load.Then,the step is Model training and verification.Experiments show that this algorithm can make the load distribution of the distributed messaging system more balanced,and can significantly improve the availability of cloud messaging services.(3)Aiming at the scenario where cloud services are combined with virtualization technology,a load balancing algorithm for a distributed messaging system in a virtualized environment is proposed.The usage characteristics of CPU,disk and network bandwidth resources in the distributed message system are analyzed,and the resources are classified at the broker level and the host level.According to the category to which resources belong,a capacity target and a distributed equilibrium target in a virtualized environment are proposed,and a load balancing algorithm based on the replica role is supplemented,so that the application range of the algorithm is extended to the virtualized environment.Experiments show that this algorithm can enable distributed message systems to achieve load balancing at the virtual machine level and the physical host level,and can improve the reliability of the cluster and the availability of services by improving the load balancing status of the system.