Research On Low Latency Optimization Technology For RDMA-Based Storage System

The emergence of RDMA with low latency and high bandwidth and high performance storage devices such as NVMe SSD have brought severe challenges to traditional storage systems.Storage systems need to optimize their software stacks to adapt to the high performance of new devices.RDMA has a different call interface from TCP,a more efficient way to use,but also has special defects.The software stack of high performance storage system needs to be optimized according to the particularity of RDMA in order to achieve ideal I/O latency.Rational use of RDMA registered memory,interaction efficiency between RDMA NIC and CPU,thread model adopted by RDMA to process messages will affect I/O latency of storage system using RDMA.The optimization of data I/O latency in RDMA level in distributed NVMe block storage system is studied,and how to realize low latency data transmission in RDMA is discussed.This paper mainly optimizes the use of RDMA from four aspects: 1.Comparing the different implementations of RDMA network interaction model in data I/O process,and further simplifying the network interaction model through WRITE_WITH_IMM to reduce the network latency of RDMA;2.Evaluating the time cost of local memory copy,and analyzing the memory copy overhead prevalent in I/O path,reducing number of copies by reusing RDMA memory;3.Use SPDK-like thread model to reduce the delay overhead caused by thread switching.4.Modifing part of the upper application interaction protocol to use imm_data and inline to reduce the number of NIC-initiated DMAs in RDMA.By comparing the latency changes before and after optimization of each optimization measure,and comparing the merged optimization scheme with the implementation of Ceph RDMA,the results show that selecting appropriate RDMA READ/WRITE operation can optimize the network interaction process,effectively reduce the delay caused by network roundtrip and RDMA software stack;Reducing the number of memory copies on the I/O path can effectively reduce the transmission delay of large data;Using imm_data and inline reduces the delay slightly;Avoiding excessive thread switching is the guarantee of low latency of RDMA.Compared with the implementation of Ceph RDMA,the merge optimization scheme can reduce the latency of small data transmission below 16 KB by 34%~54%,and large data transmission above 16 KB by 33%.The larger the amount of data,the more the latency will be reduced.