Research And Application Of Layered Distributed Consistency Protocol

With the rapid development of digital economy,the value and importance of distributed storage technology has become increasingly prominent.The storage standardization of distributed system and the multi-copy backup mechanism make it more available and expandable than the original centralized storage system.The replication technology with data consistency as the core is the key to high availability of distributed storage system.At present,the most typical way to ensure data consistency is to adopt a consistency protocol with a leader node,such as Multi-Paxos protocol.With the popularity of big data technology and the increase of various end users,the load of distributed storage system is increasing.In this case,the leader node undertakes most of the data transmission work,and will quickly reach the limit of its own resources.Its ability becomes the main bottleneck of constraining the system performance.Therefore,it is a very challenging task to solve the data transmission bottleneck of distributed consistency protocol in a high load environment.To solve the above problems,this paper proposes the following solutions:(1)This paper proposes a load balanced high throughput consistency protocol layered Multi-Paxos protocol.In the classic Multi-Paxos protocol,client requests must be received and forwarded by the leader node.Compared with other nodes,the leader node will occupy higher bandwidth resources.This paper proposes a layered design,which divides the work of the leader node into two independent layers,and separates the data dissemination task from the request ordering task.In the data dissemination layer,client requests are distributed among nodes in a distributed manner,and the work of data transmission is evenly distributed to all nodes,making the load distribution of the entire protocol more uniform;In the request ordering layer,the leader node uses the request ID to complete the ordering task,achieving higher throughput of the system.(2)A highly available key value storage system is built based on layered MultiPaxos protocol.The system consists of four main parts: client,interface layer,consistency engine and storage engine.The key value read/write and synchronization functions of the system are tested,and the performance is compared with the key value storage system based on Multi-Paxos protocol.The experiment proves that the key value storage system based on layered Multi-Paxos protocol has higher read/write performance.