Network Transfer Optimization Of RDMA-based Distributed File System

The rapid development of high-performance memory devices such as NVM hasbrought a significant improvement in data access performance for distributed file systems.At the same time,it has also brought new requirements to the system's data processing ability and network transmission ability.To this problem,RDMA is a suitable solution,which can provide better network transmission capability with less computing resources.However,at present,the related work only uses RDMA transmission as the network transmission module of the system,and does not make use of the characteristics of RDMA to integrate and adapt it effectively with the data processing of file system.This leads to the strict isolation between the network transmission layer and the logic processing layer of distributed file system,and the mismatch of the data processing ability between the software and the high performance hardware.which makes the software part of the system a bottleneck and limits the performance of hardware devices.Aiming at the objective requirement of distributed file system for RDMA network transmission in high performance hardware environment and the problem that related works can not effectively coordinate the system software layer and network layer,a new RDMA distributed file system network model is proposed.The model combines the software processing with the network transmission of the system through RDMA transmission.In terms of RDMA network memory mechanism,by maintaining the memory registration area and dynamic expansion,the system overhead of memory management in RDMA network transmission is reduced while ensuring the low latency of RDMA memory communication mode.In terms of file data transmission,it reduces the number of data replication,and uses client one-sided I/O mode for transmission to achieve high throughput data transmission.In terms of system metadata transmission,it adopts a fast receiver identification metadata transmission mode,which achieves efficient metadata transmission with low latency.The experimental results show that the distributed file system using the networkmodel has higher data throughput and lower transmission delay than the system using the original model.The throughput of data below 16 KB has increased by 12% to 74%,especially in small data block transmission scenarios,where the throughput of 1KB data has increased by 74%.Moreover,the operation efficiency of the system metadata has also been significantly improved,which has more than doubled under different concurrency levels.On the whole,the design of the network model alleviates the bottleneck effect of the software part of the system in network transmission,gives full play to the advantages of RDMA transmission and high-performance hardware,and improves the performance of distributed file system.