Research On Messaging Middle Used In Distributed Stream Computing System

With the father development of the Internet, information browsing, searching and e-commerce need to be processed online. Faced with a flood of online real-time data, such as news headlines, popular microblog, etc, these message often need to be processed immediately for use. To deal with these real-time data stream is distributed computing systems. The quality of the system will directly affect the flow, and therefore low-latency, high-throughput system has become the core indicators. Meanwhile, the messaging communication structure determines the performance of the distributed computing system, including network latency, message throughput, system scalability, and the other core indicators. In this case, how to design a low-latency, high throughput and good scalable messaging system to meet the diverse needs of the real-time data processing scene, becoming the difficulty of distributed stream computing system.Faced with this problem, this article deeply analyzed the needs of distributed stream computing system, designed and implemented a configurable and scalable messaging middlewae KMQ. It can meet the requirements of low latency and high throughput in highly concurrent distributed environment. Meanwhile, with the help of good programming interface of KMQ, developers can get rid of the traditional use of socket network programming problems and do not need to care about the underlying development of complex details. Then developers can have more time focus on the business logic of the development and improve the efficiency of web application development, reduced system maintenance and development costs. KMQ use reactor and epoll event trigger mechanism to design the atchitecture, with the aim to enhance the efficiency of the distributed stream computing system in high concurrency environment. Meantime, KMQ designed buffer for application layer and realized message codec based on Protobuf. At the same time, KMQ use a variety of techniques to optimize the system performance, such as thread pool, memory pool, configuration management, to improve system performance and scalability. On this basis, KMQ design request-response, publish-scribe and pipeline interface for the developers and conducted the function test, confirmed that it is possible and potential to use KMQ to enhance the performance of the distributed stream computing system.