Research On Metadata Management In Large-scale Distributed Storage Systems

Nowadays,large-scale data sets are typically organized and stored in distributed file systems,satisfying consistency,availability,high capacity,and performance.Based on distributed file systems,this paper mainly focuses on the problem of optimizing metadata management component.The behavior of metadata server(MDS)cluster is critical to today's large-scale distributed file systems.However,it is still a big challenge to design a metadata management scheme satisfying locality and load balancing simultaneously.Traditional metadata management schemes,such as subtree partitioning and hash-based mapping scheme,could only satisfy either locality or load balancing partially.To improve the performance of data retrieval of metadata server cluster,we try to design a metadata management scheme that has a good trade-off between locality and load-balancing.In this paper,we propose a distributed two-layer namespace tree partitioning scheme,D~2-Tree,to improve the performance of metadata management component in large-scale distributed file system.The main idea of D~2-Tree is that we firstly partition the namespace tree into two layers,i.e.global layer and local layer,then replicate the global layer to each server to maintain the load balancing of MDS cluster and allocate subtrees in local layer among the cluster to retain the data locality.To approve the validity and efficiency of D~2-Tree,we conducts thoroughly theoretical analysis and extensive experiments in Amazon EC2platform.The results show that D~2-Tree not only maintains a good trade-off between locality and load-balancing,but also has a better performance than subtree partitioning schemes and hash-based mapping schemes.