Research On Metadata Management In Large-Scale Storage Systems

With rapid development of new storage technologies as well as people's increasing demand for storage applications, network storage system is changing with each passing day. Storage system is headed towards a large storage capacity, high bandwidth, high reliability and ease of management. File system's metadata management becomes a key research site in the designing of large-scale mass storage. It has an important impact on mass storage system performance, security, availability, and scalability.Based on some analyses and researches of the bottleneck in mass storage metadata management, this paper constructs a metadata server cluster architecture model with hierarchical structure, and then designs a metadata distribution algorithm with hierarchical directory hash. Based on the architecture and the hierarchical metadata distribution algorithm, some studies are made on the mass storage system scalability, load balancing and other issues.The paper is organized as follows:On the basis of analysis to traditional MDS architecture, the MDS cluster is divided into storage management servers and index management servers from the angles of data access efficiency and management flexibility. From the perspective of function, the index management server plays a role of MDS of MDS. This MDS cluster architecture potentially brings higher performance and better scalability. It not only conducive to distribute and access metadata of file system in cluster, but also makes a platform for other researches on mass storage system applications.A hierarchical directory hash (HDH) algorithm for metadata distribution is proposed here based on mass storage system architecture. HDH is an improved hashing algorithm. Aimed at improving access efficiency of metadata, it introduces the concepts of directory storage unit (DSU) and DSU index management server. By making a directory hash to network file system, HDH algorithm distributes metadata in MDS cluster with directory granularity. This can overcome many disadvantages of scalability and load balance in traditional file hash algorithms, and in the same time provide a flexible metadata distribution management and better system scalability. The traditional file hash metadata distribution algorithm has some technical flaws in size scalability and performance scalability in mass storage system. For example, expanding system will generate lots of additional metadata requests and data migrations which will reduce the performance of storage system. HDH algorithm employs DSU index management server (DIMS) to improve flexibility of metadata management. This will reduce operation complication, diminish the additional load and increase operational efficiency. In several kinds of typical operations which may bring scalability bottleneck, such as changing directory name, deleting directory, modifying directory access permission and adding a MDS node, etc, HDH algorithm has a much better behavior in system scalability than traditional metadata distribution algorithm based on file hash.After discussing the characteristics and performance of metadata operations for load balancing problem in distributed storage system, the paper introduces a formula to evaluate the load status of storage node, and proposes a hierarchical dynamic load balancing strategy (HDLBS) to adjust load distribution among the nodes in MDS cluster. HDLBS sets a load balance management server in MDS cluster, which adjusts load distribution of MDS nodes according to load status collected from DIMS and keeps the whole system in the dynamic and continuous load balancing state. Thus improves the whole performance of storage system.