Design And Implementation Of Distributed Key-Value Storage System Based On RDMA

With the advent of the Internet + era,traditional industries and the Internet combined with each other,and Web2.0 social networks and business finance have a lot of dynamic data,which is the traditional RDBS(Relational Database Server)brings a major challenge.As a non-relational data caching / storage system,distributed key-value systems are widely used in Internet + systems.Compared with RDBS,they have high performance,high scalability and ultimate consistency.However,in the face of a large number of dynamic data and the number of clusters,network performance overhead has become a performance bottleneck of distributed key-value systems,because the system from the card to the kernel and then to the application system to go through many levels.In view of the above problems,this paper is widely used in high-performance computing InfiniBand,the main work and innovation are as follows:First of all,in the Memcached and Redis to retain the original BSD Socket socket communication module,combined with InfiniBand design IB Verbs mixed primitive network module,the client uses IB WRITE to send commands and data,so the server can use less Of the CPU to participate in,the client does not expose their own memory address space to the server side,access to lower latency and higher throughput.The server sends data or commands to the client with IB SEND.Designed to achieve the InfiniBand RDMA registered memory pool and thread pool,to RDMA registered memory pool reuse and reuse of thread pool and reduce the RDMA area for each registration overhead.Secondly,with the decrease of network delay,the network is no longer the bottleneck of the distributed key-value system.Instead,the overhead of the application itself is highlighted.We find that the traditional BSD Socket application receives the network data later Copied to the data storage memory,which does not apply to InfiniBand.So we will be the previous communication memory and data storage memory fusion,to achieve a part of the fusion system InnerCache and fully integrated NewCached system.InnerCache draws on the operating system cache mechanism,to achieve a cache-in-cache,both in the cache cache.NewCached integrates the communication area with the storage area and is divided into two parts after the integration.One part is the sparse area where the RDMA can read or write,and the other is the RDMA read-only dense area.In order to high concurrency,we designed and implemented the RDMA protocol on the communication protocol.Finally,throughout the cluster structure,Memcached uses a proxy node or mcrouter node,which runs a consistent hash algorithm that provides very good scalability,but with a large number of client-side access,the client first accesses the proxy node Access to the corresponding Memcached node,so that the agent node in the face of high concurrency is likely to become the bottleneck of the system.We design a more suitable Shinding distributed key-value system cluster structure for InfiniBand,which is a non-central structure,the client can connect to any node arbitrarily,nodes and nodes share their RDMA area,and can be dynamically adjusted according to the server load the client to connect to another server node.