Research On I/O Path Optimization Of Distributed Block Storage Based On RDMA And NVMe

The emergence of new storage and network hardware has made it possible for storage systems to pursue higher performance.For low-level devices,NVMe(Non-Volatile Memory Express)SSD(Solid State Drives)can reach eight hundred thousand IOPS,and RDMA(Remote Direct Memory Access)technology has been able to achieve latency of less than two microseconds and up to two million message per second.Most of the existing distributed block storage systems are designed for traditional hardware and cannot give full play to the performance of new hardware.Redundant protocol software stacks,context switching,and lock contention overhead on the I/O path have become the main bottlenecks that limit the performance of distributed block storage systems.In response to the above problems,a series of tests and researches are carried out,and I/O path optimization technologies for RDMA and NVMe oriented distributed block storage systems are proposed and applied to the prototype system Flame.In response to the lowlatency application requirements of storage systems,a user-mode local storage engine Light Store,which is suitable for distributed all-flash block storage systems,is designed using user-mode NVMe drivers,which reduces the overhead of redundant software stacks.Light Store further integrates the RDMA network to realize the zero-copy feature of the I/O path.In response to the application requirements of multithreading and high concurrency,the factors affecting the performance of multithreading are analyzed and found.A thread model based on asynchronous polling batch processing is proposed to process network and storage requests.And a Thread-Aware RPC(TA-RPC)scheme based on RDMA is proposed,which eliminates context overhead and lock contention overhead in multithreading.The fio tool is used to conduct detailed tests on the distributed block storage prototype system Flame.The test results show that the lightweight storage stack Light Store reduces the single-channel delay of the system by about 50% compared to the optimization before,and the dedicated I/O path model under multi-threading increases the RPC communication rate of the system by 0.93~1 times compared with that before optimization.At the overall system level,compared to the current open-source distributed block storage systems Ceph and Sheepdog,Flame reduces latency by 46.5%~63.5%,and increases IOPS by 2.5~6.35 times.