Design And Implementation Of Distributed Small Object System Based On Persistent Memory

In the field of storage system,the existing distributed storage system,limited by the traditional hardware facilities,can only make a trade-off between the total amount of system storage,the number of data stored and the overall performance of the system to meet the needs of different storage scenarios.For example,HDFS for large file scenarios has higher system capacity,but it can not store a large number of small files.Small file oriented TFS can store a large number of small files,but it can not provide high read performance.However,the increasingly active fields such as artificial intelligence,Internet of things and cloud computing urgently need a distributed high-performance storage system that can meet the needs of massive and small-size data storage.Therefore,this thesis designs a distributed small object storage system based on persistent memory,which has been launched in 2019,and describes the theoretical basis,design ideas,implementation scheme and test analysis of the system in detail.The specific work of this thesis is as follows:1.This thesis investigated the traditional distributed storage system and its basic theoretical knowledge in detail.The distributed cluster management scheme of weak center is selected to take into account the complexity of system management and the lateral expansion ability of cluster.In order to solve the problem of single point of failure and provide high availability service,the weak central node adopts the cluster scheme of one master and multiple slaves.2.Through the detailed investigation and analysis of the characteristics of persistent memory,the support of platform and operating system for persistent memory,and the file system of persistent memory,this thesis proposes a secure hardware level data hierarchical storage model.In this model,DRAM is used as the transit space to receive network messages,and persistent memory is used as the storage space for all hot data and most of the management data.The disk with large capacity in the bottom layers are used as the storage space of cold data.3.The system makes full use of the large capacity of persistent memory and introduces the global fine-grained index mechanism to realize the fast location of cache and disk data.4.Through the adaptation of the persistent memory programming model and the customized data structure of the persistent memory,the system realizes the real consistency of the data in case of failure.Finally,this thesis tests the function and performance of the system.The test results show that the system can meet the high-performance storage requirements of a large number of small-size data,and the system can ensure the high availability of services,high reliability of data and high horizontal expansion ability of the system.