Research And Implementation Of Metadata Management Technology In A Distributed Storage System

With the rapid development of networks and information technologies,global data is growing.In the face of the current PB-level massive data storage requirements,traditional storage systems have bottlenecks in capacity and performance expansion.Distributed storage systems have been widely studied and applied because of the characteristics of massive data storage,high scalability,high performance,high reliability,and high availability.According to statistics,more than half of system operations in mass storage systems are metadata operations.Metadata operations have become a serious bottleneck that limits the performance and scalability of distributed storage systems.Therefore,the research on metadata management in distributed storage systems not only follows the current development trend,but also has important application value.Based on the research and analysis of the existing metadata management model,this thesis designs and implements a metadata management technology in a distributed storage system.The achievements of this thesis are as follows:1.An overview of the existing metadata management techniques is provided.The metadata management model is divided into three types:a centralized metadata management model,a distributed metadata management model,and a non-metadata management model.Each of them elaborates the principle,gives the corresponding metadata management model diagram,enumerates several corresponding storage systems,and analyzes the performance defects of each.At the same time,two kinds of fault-tolerance mechanisms in distributed storage systems are analyzed,and their advantages and disadvantages are analyzed.2.A distributed storage system named ZettastorDBS is designed.The system oriented to the field of operators has high reliability and high scalability.The overall architecture is divided into two major modules:the data path module and the control path module,giving the data path module,access diagram,the control path module and the basic control flow composition diagram.Based on the metadata management requirements,the contents of the basic metadata management process are studied.3.Metadata management functionality in the ZettastorDBS system is implemented.First,the organization of metadata and the design of the table structure in the metadata database are given.Secondly,the design of each state of metadata and the process of migration between states are introduced in detail.Finally,the core program in the metadata management program is introduced.The design and implementation process are given,including two timers and a metadata reconstruction interface.At the same time,the entire system is deployed and functionally tested in the Linux environment.The test results show that the metadata management part can fully meet the user's requirements for storage services in all aspects of metadata management.