Design And Implementation Of High Performance Message Service On Cloud Platform

With the rapid development of Internet technology,the stickiness and dependence of various industries on Internet technology are increasing day by day.With the development of the industry,more and more rely on information and digitalization to empower traditional industries,and the volume of business content is increasing.In particular,more and more application scenarios require message queues for decoupling and peak shaving.Although traditional message queues can be expanded locally,services need to be expanded flexibly in high-concurrency and high-traffic environments.The service deployment granularity of existing traditional message queues cannot meet the flexible expansion requirements in large-scale business scenarios.To this end,based on the cloud-native concept,this thesis researches and implements a set of high-performance,low-latency,and loosely coupled cloud message queue services to meet the needs of cloud message queue in the context of cloud computing.Aiming at the cloud-native architecture of message queue,by analyzing the operation mode of the current mainstream message queue,and evaluating the existing problems after these single-point applications go to the cloud,it is found that the current stage of message queue going to the cloud is in terms of deployment granularity,cluster scaling,The distribution strategy still needs to be improved.In response to the series of problems raised above,this thesis is based on the Pulsar message queue design pattern,uses Java as the programming language for each functional module,and develops service logic based on the SpringBoot framework.Starting from low coupling and high availability,it is designed and implemented in the Kubernetes environment.CSS(compute-storage separated)cloud-native message queue.To solve the problem of service expansion efficiency,a service architecture with service logic and storage separation is adopted.Based on the stateless service feature of the HTTP protocol,the service layer is designed and implemented,and combined with the asynchronous REST dispatcher and the Istio Envoy proxy,the stateless service layer under the cloud native application is formed.This architecture that separates services from data enables services The scalability has been greatly improved;in terms of persistent storage,since persistent data is independent of application services,data migration is not required when the cluster expands or shrinks.In terms of storage implementation,it refers to the design pattern of Concurrent Hash Map with Segment as the storage unit,which improves the scalability of the application to a greater extent.Aiming at the problem of service efficiency optimization,based on Kafaka’s Sticky partition allocation strategy,resources are abstracted in the logical processing unit of the service,and the message queue data structure and processing mechanism suitable for message accumulation and the partition allocation mechanism under the cloud message queue are realized.Based on the automatic expansion and contraction mechanism of the ARIMA-SVR model,the service efficiency and hardware resource utilization of the system are enhanced.Finally,this thesis deploys a set of CSS cloud-native message queues based on the Kubernetes environment on the cloud platform,and integrates various functional modules of Istio,conducts a comprehensive test on the functions and performance of the CSS queues,and compares it with the existing Some techniques are compared.The results show that the queue can meet the original intention of the design,and has good availability and concurrency.