The Research And Implementation Of The Distributed Key-Value Storage Engine

With the development of the cloud computing and big data technology,many massive-scale data applications start to emerge.These new applications take the more and more high request to the data storage.However,the traditional relational databases,such as Oracle and MySQL,have become increasingly difficult to meet the data storage needs of the cloud computing environment.Distributed non-relational database fully combines the anti-single-point-of-failure capability of distributed systems and natural scaling-out characteristic of NoSQL.It can easily cope with the challenges of the massive data storage problem.For this reason,this paper designs and implements a distributed Key-Value storage engine.And based on the study of distributed transaction and storage performance expansion,we do the following specific researches:(1)Based on the existing distributed theory,this paper studies the related technologies of distributed storage,introduces the concept,characteristics and related theories of distributed NoSQL database,and analyzes theadvantages and disadvantages of the architecture of the widely used NoSQL storage system.On this basis,we design the architecture of the distributed Key-Value storage engine MiniKVDB.(2)Based on the commondistributed transaction model and the commit protocol,we design the distributed transaction solution for MiniKVDB.In the module of the data replication,we deeply analyze the code structure of the Raft consensus protocol.Based on the long fat pipes and sliding window mechanism in the TCP protocol,we add the buffer data area in the log module of the replicated state machine.The experimental result shows that the improved consensus protocol can reduce the communication load and the frequency of reading and writing disks so that the distributed systems can have high availability and consistency.(3)Based on the Raft consensus protocol,we design the mechanism of the flexible scalability and data-sharding strategy of MiniKVDB.We add the ShardMaster node as the cluster management module to store the meta information.ShardMaster will move the shards between nodes according to the scheduling algorithm and the load performance of each node in the cluster.(4)Based on the above-mentioned theory,we develop and implement a distributed Key-Value storage engine MiniKVDB.The development language is Golang.In MiniKVDB,in order to ensure the high availability,we deploy the improved Raft consensus protocol to achieve the strong consistency of data replication.We also deploy data-sharding policy to balance load and data automatically.At last,we make a detailed test of the storage engineand propose the improvement plan in the future.