Design And Implementation Of Metadata Management In Distributed Small Object Storage System

With the rapid development of computer hardware and software,the types of data we come into contact with in our work and life are becoming more abundant,such as documents,documentaries,short videos,pictures,etc.The storage demands of these data are gradually increasing,and the efficient storage and access of short videos,pictures,small videos and other data in chat software is a problem that needs to be solved at present.Existing object storage systems are mostly used to store unstructured types of data.In actual business interactions,most operations require access to the metadata management server,and the processing speed of metadata affects the efficiency of the entire system to a certain extent.Due to the small size of unstructured data,the frequency of access in actual storage is relatively high.In order to achieve efficient storage and access to this type of data,this article has carried out related work on the metadata management of massive distributed small objects.The specific research content in the paper is as follows:1.Highly available metadata management architecture.Distributed clusters are used to realize the management of metadata,and the Raft protocol is used to synchronize the state between the metadata nodes to achieve data consistency.The leader is selected among the nodes through the Raft protocol,and the leader completes the related work in the cluster.The other metadata management nodes are used as backup machines.When a node is abnormal,the system can still serve the external normally.2.Dynamically adjust the system based on node load to achieve resource balance among nodes.The metadata management server manages the mapping relationship between the hash slots and the service nodes,allocates a certain range of slots for each node according to the memory size of different service nodes,and evenly maps the service nodes to these slots.When the user initiates a data operation request,the metadata management module will select the data node to which the corresponding slot belongs according to the hash value of the requested data file,and the data node will complete the subsequent tasks.3.In view of massive access to small object metadata,in order to reduce the network overhead of metadata cluster leader accessing the underlying object storage,a cache is set up in the cluster,and the read efficiency of hot data is improved based on the LRU algorithm.4.Using two data redundancy recovery technologies: multiple copies and erasure codes,the corresponding fault tolerance method is selected by the number of tasks requested by the user,so as to ensure that the data is not damaged and the reliability of the entire system.Through the above work,this paper has completed the design and development of metadata management,and carried out experiments and analysis from the functions and performance of the entire system.The results of the experiments have proved the high reliability and high availability of the system,indicating the feasibility of the proposed scheme in this paper.