Research On Distributed Key-Value Storage Engine Technology

In the context of the rapid growth of data scale,research on distributed databases is becoming an important issue in the current storage field.The research group attempts to build a distributed relational database to explore the technical principles of distributed storage,and plans to use it to improve the performance of the key server cluster of the research group’s cloud encryption system in the future.To this end,a distributed Key-Value storage engine is first constructed to support the underlying data storage,and then the SQL layer will be built on this basis to complete the distributed relational database.This article discusses the technical research in the process of building a distributed Key-Value storage engine called SimpleDKV.The architecture design of SimpleDKV divides the system into two parts:storage module and central configuration and scheduling module.The system slices the stored data.The storage module uses SimpleKV Group as the basic unit.Each SimpleKV Group is responsible for a certain number of data slices,and then provides clients with efficient storage services for these data.Each SimpleKV Group consists of an odd number of SimpleKV nodes.The lower layer of SimpleKV is the Raft module used to reach a distributed consensus within the group,and the upper Raft-based state machine service updates the state according to the Raft log and processes client requests.The central configuration and scheduling module consists of a single Scheduler Group composed of an odd number of Scheduler nodes,and also uses Raft to reach a distributed consensus within the group.The Scheduler will dynamically adjust the system configuration and resources according to the information reported by each node and thus improve the resource utilization efficiency of each service node and use load balancing to improve the efficiency of the client requests.The implemented Raft has carried out some engineering optimizations,and realized many optimizations including fast rollback,request batch processing,synchronization of empty logs,asynchronous execution,and snapshot functions.These optimizations can speed up the processing of upper-level state machine services.Scheduler and SimpleKV are both built as Raft-based state machine services to greatly improve system performance.