Optimization And Implementation Of Data Transmission Mechanism Based On RDMA

With the rapid development of “internet +” era,the promulgation of national big data strategy and the advent of artificial intelligence era,the major internet companies are aware that the data would become the future oil and they have started to get a lot of data from people's basic necessities of life.The widespread use of new hardware devices(NVMe SSD,RDMA-enabled NIC,3dXpoint,etc.),there performance(magnitude,delay,etc.)is better than before.At this time,the time overhead of traditional software layer is highlighted.As the most popular distributed processing system in the industy,Apache Spark is widely used in data analysis,interactive data query,machine learning and other fields.As a distributed system improved on MapReduce,Spark's Shuffle process involves very time-consuming network IO and disk IO.So the performation of Shuffle is currently one of the bottlenecks in determining the overall performation of Spark.At present,the 25 Gb Ethernet that supports the RDMA protocol has been deployed on a large scale in the major domestic internet companies.So it is a significant research to explore the deep integration of RDMA technology and Spark.With the Infiniband architecture that is commonly used in high performation computing,this paper's work and innovation as follows:The memory that store the data transmitted by RDMA NIC,must be firest register the memory metadata to the RNIC.The registration operation needs to go through the PCIE bus,so this is a time-consuming operation.In this papter,we refer to the current mainstream memory allocator,and design and implement a hierarchical RDMA memory pool based on the Boost Pool.This memory pool could reuse RDMA memory,reducing the cost of each registration operation.Expermental results show that the memory pool can be well adaped to multithreaded competition and achieve higher efficiency.RDMA offers a variety of modes and opetations of data transfer,all of them have different strengths and weaknessed and are suitable for different application scenarios.In this paper,We divide the data into small and large data block according to a certain threshold.For small data blocks,usd RDMA write_with_imm operation to directly wirte data to a designated remote memory area.For large data blocks,first use RDMA write_with_imm operation to data metadata,then use RDMA Read operation to transfer data to remote memory according these metadata.This data access mechanism uses different transmission modes based on different size data,effectively taking into account commonality and efficiency.In the foregrounp of above,this peper designed and implemented a common high performance RDMA network library — baiyun.With reference to netty in use-oriented prigramming models,baiyun is characterized by its simplicity and ease of use.It is based on open-source project brpc,which provide user-mode thread and execution queue,to achieve high performance and universal goals.Finally,this paper optimizes Apache Spark's Shuffle mechanism based on baiyun.The main idea is replace the network module in Spark Shuffle with baiyun via JNI,witch achieves relatively high efficiency by pipelining operations.A large number of experimental results show that the optimized Spark's performance has been grealy improved.