Optimization And Implementation Of Layered Consensus In High Load Environment

With the emergence and rise of big data,the value and importance of distributed storage technology has become increasingly prominent.Compared with the traditional centralized storage system,distributed storage system has the advantages of low cost,easy expansion and high availability.Replication technology is the key to realize the high availability and fast response in wide area of distributed system,and the performance of data consistency is an important indicator of replication technology.At present,the most effective solution to the problem of data consistency is the use of leader based distributed consistency protocol,such as the typical Multi-Paxos protocol.However,with the popularization and widespread application of big data technology,the distributed storage system is more and more loaded.In this case,the ability of the leader node constrains the improvement of system performance to a large extent.How to solve the bottleneck problem of Multi-Paxos protocol in high load environment is a very challenging work.To this end,this dissertation aims to improve the throughput of system,and to study the optimization problem of distributed data consistency protocol by widening the bottleneck caused by the leader node in Multi-Paxos protocol.The main research contents and innovations of this dissertation are summarized as follows:First,we theoretically analyze the bottleneck problem of the leader node in MultiPaxos protocol,and on the basis of this,propose an improved layered Multi-Paxos protocol.The proposed scheme is capable of distributing the requests responsible for the leader node in the Multi-Paxos protocol to all storage nodes,so that the leader node is only in charge of ordering the IDs of the requests,by which the load on the leader node can be reduced effectively,and the resource utilization the system can be improved.Second,we introduce the idea of batch processing and pipeline into the layered Multi-Paxos protocol proposed.Then,we deduce the quantitative relationship between batch processing and pipeline in the proposed model,and then optimize the parameter setting of batch processing and pipeline according to the derived results.Simulation results show that the proposed scheme can effectively improve the ability of parallel processing and thereby improve the overall performance of the system.Last but not least,in order to verify the correctness and validity of the proposed scheme in real environment,we launch experiments of the proposed Multi-Paxos protocol and its optimization schemes.The experimental results are in accordance with the simulation results,which prove that the proposed scheme can not only solve the bottleneck problem of the leader node effectively,but also outperforms the existing scheme in term of the number of responses per second.