Research And Implementation Of Communication Protocol Based On RDMA Across User And Kernel Space

With the explosive growth of data and the popularity of Infiniband and other high speed network technology, the high performance distributed file system provides an effective support for large data storage.The traditional TCP/IP network architecture has become a bottleneck,which is difficult to make full use of network bandwidth,. RDMA(Remote Direct Memory Access) technique is adopted to directly read and write to the remote memory, avoids the overhead of duplicated memory copy, to meet the high-speed communication needs in the distributed application.In the distributed file system, in order to ensure higher read and write performance, the client is usually implemented in the kernel space, while the back-end storage server is implemented in the user space for the purpose of simplifying the design, and this will arise the problem of communication across user and kernel space.It uses Remote Procedure Call(RPC) model based on TCP/IP protocol stack, even if the hardware using 40 Gbps high-speed Internet provided by Mellanox Infiniband company, due to the limitation of the software stack communication architecture, network bandwidth can not be fully utilized.Therefore design a set based on RDMA application in network communication architecture, which achieves RPC over RDMA communication interface across user and kernel space.In the implementation of communication architecture across user and kernel space, the client in kernel space is based on the NFSoRDMA state machine model, and the server is accomplished by RPC over RDMA in user space. In accordance with the logic of the traditional RPC achieve the the second half of the call, coding and decoding process on client and server combined with RDMA primitive operation,which implements a scheme of accelerated distributed systems communication across user and kernel space, which is suitable for the distributed application scenarios of efficient communication between the user space and the kernel space.Comparative test results show that the RPC over RDMA communication architecture design makes network bandwidth utilization rate improving significantly, nearly 50% increase of the peak bandwidth,which makes full advantages of Infiniband network.