Research And Implementation Of User-level File System Based On RDMA And NVM

Efficiently accessing remote data is a challenging problem for data processing systems,where network and storage are the main performance bottlenecks.Meanwhile,the development of technologies in non-volatile memory(NVM),in-memory file systems,and RDMA networks provide new opportunities for solving the problem of remote data access.However,the widely-used distributed systems can not fully exploit the high-performance characteristics of NVM and RDMA.Specifically,they are designed for traditional Ethernet and HDD/SSD devices.The network layer is separated from the storage layer in their architectures,which results in high software overhead.Additionally,it is extremely challenging to design a high-performance distributed file system based on NVM.The reason is that the latest researches are not clear about the characteristics of real NVM devices,only regard NVM as DRAM that provides data persistence,then use DRAM to simulate NVM.Currently,the Intel Optane DC Persistent Memory(Optane DCPM)is the only NVM device connected to the memory bus via DIMMs currently available on the market.However,there are few works exploring the characteristics of Optane DCPM.In this case,some real characteristics of Optane DCPM,such as the difference between its access characteristics and DRAM,are not discovered.Aim at above problems,we have some works as follows:1.Using Optane DCPM,we explore several architectures for accessing remote NVM and compare the performance of these architectures with existing architectures,including NVM over TCP,DRAM over RDMA,local access to NVM,and local access to the DRAM.The experimental results show that the characteristics of NVM are completely different from those of DRAM.And the high potential of NVM for high-performance,remote in-memory accesses can be revealed with careful design.2.Based on the above experimental conclusions,we find that the traditional distributed file system can not fully exploit the high performance of NVM and RDMA.And the current persistent in-memory file system based on simulated NVM by DRAM can not represent the actual performance of NVM.According to the above conclusions,we design and implement a user-level in-memory file system,RIMFS(Remote Inmemory File System),based on NVM and RDMA.RIMFS consists of clients and a server.In the server,the file management module and the memory management module are designed to manage the data stored in NVM at a low overhead to prevent the software layer from becoming a system bottleneck.In the client,the RIMFS library encapsulating RDMA communication is provided to the external process.And the external process sends file system instructions to the client through the RIMFS library function to access the file data stored on the server.In RIMFS,the whole data access process is carried out at the user level to avoid the switching between user-level and kernel-level,which effectively reduces the number of memory copies and the overhead.3.To further improve the performance of RIMFS,we design the metadata caching mechanism and the metadata prefetching mechanism.These two mechanisms can effectively reduce the latency of I/O and prevent the network transmission latency from becoming the bottleneck of small-grained I/O.By comparing against the DAX-supported Linux file system,Ext4-DAX,we show that RIMFS can significantly exert the performance potential of NVM and RDMA.The performance of remote reads on RIMFS over RDMA is 11.44 × higher than that on a remote Ext4-DAX on average.The experimental results also show that the performance of remote accesses on RIMFS is maintained on a heavily loaded data server with CPU utilization as high as 90%,while the performance of remote reads on Ext4-DAX is significantly reduced by 49.3%,and the performance of local reads on Ext4-DAX is even more significantly reduced by 90.1%.And the performance comparisons of writes exhibit the same trends.