The Research And Implementation Of Performance Optimization For In-memory NoSQL System Redis

The rapid development of Internet technology is accompanied by the emergence of massive data,various scenarios require greater performance for large data storage and management systems such as reducing latency,increasing throughput rate.NoSQL system is one of the important components of the process of big data processing because of its high scalability and efficient processing of half-structured or unstructured data.Various NoSQL systems are geared towards the needs of different upper-level applications,provide a large number of configuration parameters which manage the system itself and the resources used by the system,and setting the configuration parameters to suit the current scenario is one of the key means to tune the performance of the system.As one of the most popular NoSQL systems,Redis is a kind of open source high performance in-memory database which supports data persistence.It is worth considering how to tune the Redis to help the user get the maximum performance from the system according to the specific application scenario.Redis,as an in-memory NoSQL system,saves all data sets and intermediate results in the main memory system,which makes network latency,network bandwidth and CPU utilization become the major performance bottlenecks.As one of the common RDMA techniques,InfiniBand is frequently used in high performance computing because of its high bandwidth and low latency.It is worth considering how to combine it with Redis.Using RDMA technology over InfiniBand can efficiently improve network-communication's performance,increasing throughput and reducing network latency while reducing CPU utilization.The main work are as follows:(1)Analyzed the characteristics of Redis,and based on Redis performance metrics to tune the performance of the system.Redis is popular because of open source,and its use often relies on experienced database administrators to configure the system's operating parameters to get efficient system performance.For novices who do not have the experience of Redis system operation and maintenance,they can only use the default configurations to meet the most basic functional requirements.Therefore,as for Redis,in order to help more users who use it to get better performance;we analyzed the factors affecting redis performance,and gave tuning methods,then summed up the tuning rules and designed a rule-based parameter management system.(2)Examine and evaluate the two message transmission semantics provided by InfiniBand.In order to reduce the delay of network transmission and improve the real-time access performance of Redis,the network delay is optimized by InfiniBand which is one of the RDMA techniques.Because InfiniBand provides two kinds of transmission semantics which are send/receive and RDMA.Different transmission semantics provide different settings;different transmission semantics have different performance in various communication modes.Therefore how to choose the most suitable transmission semantics and settings is a problem to be considered.In this paper,the performance evaluation and comparative analysis of the two kinds of message transmission semantics are carried out,and then the choice of transmission semantics and settings is decided based on the obtained test results.(3)On the basis of performance evaluation of InfiniBand transmission semantics,RDMA is used to optimize the performance of Redis,which can effectively reduce network communication delay and improve system throughput.In order not to damage the original communication module of Redis,we propose a replaceable communication model framework,so that the optimized Redis can support both the Socket based communication module and the RDMA based communication module.We implement the communication module based on RDMA,evaluate and analyze the time delay.In RDMA based communication module,we proposed a dynamic Memory Region allocation method avoiding memory waste caused by allocating too much buffer at one-time,efficiently save the memory space;and we designed a client request polling method to help the Redis server check the receipt of client requests and complete a large scale data fragmentation method.In general,the Redis using the RDMA based communication module is 8 to 10 times faster than the Redis using the traditional IPoIB based network communication module;our design indicates that when value size is equal to 3KB,the SET operations of RDMA based Redis is faster than IPoIB based Redis about 2 times.