Distributed Metadata Organization Method Based On File Path And Attribute Information Separation

With the advent of the era of big data, storage systems for big data have increasingly emerged. The centralized metadata management systems are more and more overloaded and will become the bottleneck of the big data storage system with the growing amount of data. Therefore, people put forward a variety of distributed metadata management methods. However, due to the structure of the metadata is more complicated, there is currently no a way to improve the performance and scalability of metadata management at the same time.This paper proposes a distributed metadata organization method, which separates the file path and attribute information. We organize metadata into two parts, the index of directory and metadata attribute information. Divide the metadata in the unit of directory or less than the directory into the different buckets by building the index of directory, and assign the buckets to different metadata servers according to the load of the metadata cluster. The method can improve the management performance of metadata by using directory index and bucket. At the same time, the scalability of metadata management can be achieved by building the directory index with the consideration of the cluster load. In addition, we propose the position-based metadata caching strategy, which solves the problem of inconsistency location of cached information and simplifies the flow of metadata management.The test results show that the proposed method has high performance, especially for high concurrency. It has a good scalability and metadata locality, and it don't limit the size of the directory. It avoids unnecessary migration caused by renaming the metadata. In comparison with the centralized metadata management method, the operation performance of creation, searching for metadata has been improved by several times, while the proposed method uses a single metadata server.