Research On Optimization Technologies Of Distributed Block Storage System Based On NVMe Over Fabrics Protocol

In recent years,with the explosive growth of global data generation,upper-layer applications have higher and higher requirements for underlying storage.At the same time,a series of emerging technologies in storage media,storage protocols,and network communications have emerged.The Non-Volatile Memory Express over Fabrics(NVMeo F)protocol extends Non-Volatile Memory Express(NVMe)semantics and is widely used in high-speed interconnection of distributed storage.However,the software stack designed by the traditional distributed storage system for the mechanical hard disk and the transmission control protocol network has a large amount of management redundancy and data replication,which cannot give full play to the role of the high-speed interconnection protocol,which has become a new performance bottleneck.In addition,in the scenario of heterogeneous storage devices,the management model of traditional distributed storage systems also lacks reasonable division of nodes and efficient allocation of resources.Aiming at the redundancy of the software stack,the redundancy of the management mechanism in the critical path of the storage system and the extra overhead introduced by multiple data replication were analyzed,and a lightweight optimization scheme of the critical path was proposed,including a memory management mechanism based on local inverse control flow optimization and data multiplexing method based on Remote Direct Memory Access(RDMA)protection domain penetration to reduce the latency of critical path access.Aiming at the management model of the system in the heterogeneous scenario of storage devices,the shortcomings of the existing storage system management models were analyzed.A Domain Management Model Based on Adaptive Partition(DMM-AP)and a Resource Management Strategy Based on Authorization and Autonomy(RMS-AA)were proposed,which improved the overall performance of the system.The prototype system Flame+ was designed and implemented,and the above-mentioned lightweight critical path and new management model optimization scheme were applied.The evaluation shows that the lightweight critical path solution increases the Input/Output Operations Per Second(IOPS)of Flame+ by 57.71%～124.28%,the bandwidth increases by 66.89%～123.77%,and the latency reduces by 33.86%～36.32% compared with the traditional solution.The DMM-AP and RMS-AA enable the aggregation performance of Flame+ to scale almost linearly and reduces the latency of user resource allocation by25.54%～31.30%.At the overall system level,compared with the open-source distributed storage systems Ceph and Ceph+SPDK,Flame+ improves IOPS by 2.36x～5.78 x and3.92x～6.05 x respectively,and reduces latency by 65.89% and 72.21% respectively.