The Research And Implementation Of Fault-tolerant Metadata Cluster Management For PB-scale Storage System

With the development of computer technology, information technology and the Internet, it makes high-performance computing, business computing, large-scale data processing, information processing and other technologies being used widely. Cluster systems become more and more favored because of their higher cost-performance and better scalability. At the same time, these applications bring forward the requirements of greater capacity, higher performance and higher availability for distributed data storage.Emerging object-based storage architecture provides unprecedented scalability and aggregation throughput by taking advantage of existing processing technologies, network technologies and storage components. It provides the foundation for the construction of a new generation's large-scale parallel storage system.In this paper, on the basis of comprehensive and in-depth understanding of object-based storage architecture and existing storage systems, it has an in-depth study for the fault-tolerance of metadata cluster management which is involved in the object-based large-scale cluster storage system, and it puts forward some new ideas and solutions. The main contributions are as follows:(1) A high availability metadata management model based on directory object replica is proposed for PB-scale storage system. By using with the high efficient and concurrent placing, updating and migration strategy and management mechanism of directory object replica, this model not only assures the directory object data reliability and availability but also improves reading aggregation network bandwidth and reading operating performance. We quantify the availability of storage system by using Markov Reward Model. The practical functional and performance testing results indicate that the model can effectively guarantee the high availability of storage system and improve the overall access performance of metadata server cluster.(2) An efficient checkpointing scheme for journal-based metadata management is proposed. By implementing checkpointing to journal-based metadata management, the system will be assured the effective storage space utilization, while achieving a rapid system recovery, the high availability of system is guaranteed too. Actual testing results show that this approach can utilize disk space fully and improve the system's recovery time, and ensure the high availability of metadata, improve the access efficiency of metadata.