Cloudqueue: An Internet-Scale Messaging Infrastructure Based On Hadoop

With the development of Internet technology and the Internet of Things technology, data showing explosive growth. In order to handle these vast amounts of data, we need to build a much larger, more complex, more flexible distributed systems, communication and collaboration between the system becomesis essential. Message-oriented middleware uses efficient and reliable messaging platform-independent data exchange, shielding the hardware and software differences between different systems, to communicate and exchange data between different systems transparently. However, the traditional message-oriented middleware is obviously has been unable to meet the requirements of the current large-scale, cross-data center communication between distributed systems. Hadoop is an open source distributed file system infrastructure, it is also a framework of distributed applications running on large clusters by the general-purpose computing devices, which can provide a stable and reliable interface for the application to build a high reliability and good scalability of distributed systems.Based on demand of the current distributed system messaging, combined with the Hadoop cloud computing platform, we design a Internet-scale, high performance and reliable message queue service platform, Cloudqueue. Via the message service provided by the Cloudqueues platform, which has store and forward, reliable transmission across the Internet and other characteristics, the Cloudqueue can provide high performance, reliable, high-capacity Internet-oriented messaging services in the era of cloud computing.Based on the analysis of the basic knowledge of messaging middleware and Hadoop key technology of the design objectives, the paper shows the overall architecture of the Cloudqueue system, and then, through the analysis of the key technologies related to the proposed technical, we design the architecture and objects model of the Cloudqueue. And then, we detail the description of the Cloudqueue implementation and any associated interface definition.Finally, through experiments and analysis we show that the proposed Cloudqueue system has higher performance, reliability and scalability than the traditional messaging middleware cluster system.