Key-Value Store Performance Optimization Based On RDMA

With the rapid development of e-commerce?social networks and mobile Internet,the Internet user community is growing along with this wave of rapid growth,while massive amounts of data are constantly being produced and brought together.These huge amounts of data pose a major challenge to traditional relational databases,and key-value storage systems,as a non-relational data storage system,have better performance and scalability.However,in the face of the continuous increase of dynamic data and the increasing scale of clusters,the performance overhead of the network has become a performance bottleneck of memory-based key-value storage systems.Due to the development of high-performance networks,data centers have gradually introduced RDMA(Remote Direct Memory Access)technology in high-speed networks.RDMA has the characteristics of high bandwidth,low latency,and light CPU load.It provides performance guarantee for applications and provides new ideas for key value storage system design.With the current technology,although the RDMA network can be directly used in the manner of IPoIB(Internet Protocol over InfiniBand)without rewriting the application,this method does not maximize the performance of the RDMA.In order to obtain the RDMA high-speed network transmission service,the key-value system needs to use the RDMA unilateral element at the application level and is responsible for managing the network transmission.In view of the above problems,this paper combines the RDMA technology in high-performance network to design and implement a high-performance,low CPU load key value storage system Chequer.The main work includes the following aspects;(1)Firstly,combined with the characteristics of RDMA primitives,the operation flow of the key-value system is redesigned:the get operation uses the RDMA read primitive to reduce the server-side load,and the CPU bypass of the server is actually implemented;The copying operation of the data,taking into account performance and workload,provides two modes of transmission.A data structure with self-checking function is designed to solve the problem of read and write competition between the client and the server.(2)Secondly,a shared hash table based on linear detection is designed to solve the problem of client cache invalidation and improve the hash hitting rate to reduce the number of read rounds of the client,which further improves the performance of the system.(3)Finally,when the system is finished,the three aspects of network module,memory management and the selection of client transmission mode are further optimized.The RDMA network transmission solves the scalability problem of the network connection by using queue multiplexing,and processes the message based on the event-driven mode,achieving high throughput and low latency.Aiming at the problem that the server may become a performance bottleneck,a buffer memory pool based on the Buddy memory management algorithm is designed and implemented to respond quickly to the requesting and releasing of the server.Thinking the load on the server side and related hardware performance,and adaptively adjust the parameters to select the appropriate transmission mode for the client's input operation.Through the comparison of experimental tests and analysis,the Chequer system has a large performance improvement.