Design And Implementation Of A High Throughput Distributed Message-oriented Middleware

The continuous development and improvement of Internet application is promoting the transformation of enterprise projects from traditional Monolith to Microservice architecture.The centralized system can be vertically divided into several small independent services according to the business characteristics,which can effectively reduce the maintenance cost of the system and improve the development efficiency of version iteration.The independent deployment of each service also brings a flexible way to adjust the cluster size,which makes the elastic expansion of the system more lightweight.The vertical division with the system is also accompanied by the communication needs of various services.Message-oriented middleware is a distributed framework that provides reliable asynchronous network communication for different systems or different modules in the same system.It receives messages from upstream services,stores them and forwards them to downstream services,which plays a connecting role in the system architecture.However,mainstream message-oriented middleware products still have problems in the actual production and use of enterprises,such as poor resistance to message accumulation,weak system load capacity,and poor elastic expansion capacity.In response to the above problems and the actual needs of enterprises in production applications,this thesis designs and implements a high-throughput message-oriented middleware Whale MQ,which can allow the accumulation of massive messages on the basis of correct message delivery,providing efficient and stable performance.The main contents and achievements of this thesis are as follows:(1)In view of the problems and defects in Kafka's design,this thesis proposes the following optimization scheme: design and implement the Meta Server of the system based on relational database,providing functions such as service registration,cluster health status monitoring and dynamic routing discovery;design and implement a message persistence model based on Message Log to balance the high-performance and stability of the Kafka,and a two-level cache structure is introduced to realize the mass storage of messages and reduce the IO times between the system and disk;a flexible message consumption model and message queue dynamic mapping mechanism are designed to improve the elastic capacity of the system.(2)Based on the communication requirements of each module of the middleware and the principle that the agreement is greater than the configuration,the lightweight network communication protocol is customized in this paper.While minimizing the proportion of protocol information in the data package,the scalability of the protocol is considered,allowing the business side to expand flexibly and realize the customization of communication.Finally,the Netty framework is introduced to realize the efficient network communication module.(3)The function test,performance evaluation and reliability test of the system are carried out.The function test is mainly focus on verify whether the basic functions of the system are correct and available;the performance test is to test the load capacity and throughput of the system in eight experimental scenarios such as asynchronous storage and asynchronous replication,and then compared with Kafka.The experiment proves that the system orders topic to send TPS(Transactions Per Second)in the two master and two slave standard deployment architecture Second)can reach 46 K,which can meet the needs of the vast majority of application scenarios,and the system performance does not change much with the increase of the number of topics,and the performance is stable;the reliability test compares the disaster tolerance ability of the system under the multi-server room deployment,and the experimental results are as expected.