Design And Implementation Of Distributed Message Queue Supporting Transactions

Since the age of internet was coming,which also brought a change from the application architecture,application of client/server model to the client/middleware server/database server three layer model transfer become Internet application trend.Middleware services meet the collection of various functions(clusters)to provide a good environment for other application services.Message oriented middleware mainly deals with burst messages and persistent message peaks,providing the ability to balance server resource load and stabilize the production environment.The asynchronous service capability of message oriented middleware has changed the problem of business layer dependency,which makes it easy for the system architecture to expand horizontally.Message oriented middleware,also known as open message service,has different implementations.Early message middleware is mostly developed by foreign technology vendors for internal commercial software,which is difficult to use,complex in design,poor in generality,and not to be open source,and can not meet the complex environment of Internet applications.Distributed transaction messages are special cases that require multiple messages to be complete and orderly.The implementation of ideas and usage scenarios leads to different design schemes of message middleware.Early distributed transaction design takes more consideration of strict transaction consistency,and does not pay attention to the optimization methods such as disassembly of transaction messages,asynchronous execution,reordering of transaction messages.At present,the rise of Internet industry in China brings forward the design scheme of message middleware.Comparing domestic and foreign counterparts with high throughput to support high concurrency of open source message middleware and have mature distributed transaction solutions,there are still no mature standards.In this paper,a high performance and lightweight message middleware DTube based on Java language is described for business diversity,uncertainty of peak value,high efficiency and concurrency,convenient use and stable service.In this paper,based on the basic principles of Linux operating system and distributed system theory,the realization of distributed storage based on message oriented middleware is designed.The main work includes the following aspects:(1)Design and implement a distributed storage model for message accumulation.This part includes the design of message classification,message storage queue,and distributed distribution of message storage.The distributed storage model uses each storage server as a file logically.Based on the distributed storage model,we can achieve the following steps:sending and receiving ordinary messages;sending and receiving sequential messages;and filtering the service side when consuming.The related theories of distributed consistency are analyzed:two armed forces,Byzantine Generals,and Paxos algorithm.Analyze the message middleware specification and refer to the advanced message middleware design ideas.(2)Design and implement a distributed transaction message processing model.Distributed transaction is based on the principle of XA two phase commit protocol.This part deals with the handling of distributed transactions at multiple stages.Based on this model,we can implement the distributed transaction processing commit at the sending side and the processing and submission of consumption message consumption.(3)Design and implement the query function for distributed messages.This part provides the function for large scale message query:querying messages through Message Id;querying messages through Message Key.The Message Id query is by recording the location of the file.Message Key query designed a file storage structure similar to Java7's HashMap.Then the principles of storing different types of messages in the message queue are explained,and simple examples and processes of sending consumer messages are sent.(4)Use DTube optimization and other implementation details.Optimization includes:file system,deployment mode,Linux system kernel tuning.DTube other aspects of the design include:network transmission and serialization,I/O zero copy,high availability backup design.Optimizing network transmission and serialization is an effective way to improve performance.