Research And Implementation Of Distributed Storage System For NVMe And RDMA

In the recent decade,the rapid development of new storage and high-speed network technologies has achieved significant hardware advances,resulting in the latency for devices significantly reduced.On the storage,the emergence of the NVMe(NonVolatile Memory Express)protocol makes it possible for the SSDs to achieve the latency as low as ten microseconds,and the throughput up to 500 k IOPS.The performance of NVDIMMs'(Non-Volatile Dual In-line Memory Modules)products is already comparable to DRAM,for example,Intel ? Optane? technology.On the network,the high-speed RDMA(Remote Direct Memory Access)technology,which was previously used in high-performance applications,is gradually becoming popular in data centers,which also allows the round-trip latency of data transmission between different nodes to be less than five microseconds.Therefore,traditional software technology has been difficult to adapt to new hardware technologies,and has become a new bottleneck for large-scale data processing applications.To solve these issues,a series of research and discussion has been carried out in this dissertation.We design and implement a distributed storage prototype system,PlutoX,supporting new hardware and supporting a variety of hybrid workloads.In this prototype,we propose a set of customized distributed storage technologies for new hardware.And,we perform a large number of experiments to evaluate the overall performance of the prototype system.The main contributions of this dissertation are as follows:(1)Proposing a customized distributed system framework for new hardware.The system should be higher internal correlation,no functional overlap,and should adopt a light-weight userspace I/O software stack,to further explore the potential performance of new high-speed hardware.(2)Supporting a programming interface for relational tables.The upper-level execution engine can perform data operations in the same way as in relational databases,so that the upper-level execution engine can quickly implement different query optimizers,further supporting OLTP,OLAP and HTAP multiple hybrid workloads.(3)Considering the multi-queue feature of the NVMe protocol,this dissertation proposes a dedicated queue mechanism for isolating read and write requests,further improving the response latency of read and write requests.At the same time,the PlutoX prototype also reduces the footprints of memory copies on the I/O path,improving the memory usage efficiency and reducing the unnecessary copy overhead.(4)Implementing a hybrid data transmission strategy.Based on two different RDMA communication models,this dissertation gradually analyzes the overhead of prototype's data transmission in many different combinations.For the different requirements of different modules,the PlutoX system adopts different data transmission methods.This hybrid approach improves the efficiency of data transmission within the distributed storage system.In summary,the new hardware technologies not only improve hardware performance,but also promote the transformation of software technology.This paper studies the issues of distributed storage based on NVMe and RDMA,designs a set of customized storage technologies for new hardware,and implements a working prototype system,PlutoX,compared with the state-of-the-art distributed storage system.