The Design And Implementation Of A Message Publish-subscribe Service Based On Kafka

Netease YanXuan is an e-commerce system designed and developed based on the microservice architecture and the system is split into microservices that can be developed and deployed independently according to the difference of business logic.Communication between services is achieved through two lightweight communication methods:HTTP and RPC.However,both of these communication methods are synchronous and point-to-point which may cause high coupling,easy blocking and no buffering of requests between services and affect the performance and stability of the entire system.In order to solve the above problem,we develop a message publish-subscribe service(MPS)which extends communication between services by providing the ability of asynchronous communication.MPS uses Kafka which is a message middleware for message storage and consumption because of its high throughput and high availability,and introduces a concept of "Logical Topic" as the basic unit of message publishing and subscription based on the "Topic" defined in Kafka.This thesis mainly introduces the design and implementation of the message publish-subscribe service and divides the service into five modules.The service access management module is responsible for managing external services that access MPS,logical topics and subscriptions to logical topics.The message producing module stores messages published to MPS in the Kafka cluster persistently.The message consumption and push module simplifies the message acquisition logic by consuming messages in the Kafka cluster and pushing them to external services actively.The message failover module provides a fault tolerance mechanism for the failure of pushing message to avoid loss of message.The caching module improves query efficiency by caching relevant data.Currently,the initial version of the message publish-subscribe service has been developed and deployed into the production environment.By accessing MPS,more and more services have been optimized on performance and invocations.