Structured Storage System The Design And Implementation Of Metadata In The Cluster

With the development of technology and the advance of informationdigitalization, more and more data need to be stored permanently, and the storagecapacity is getting bigger and bigger. How to effectively store large-scale data andprovide a common interface becomes a hot topic in the mass data age. Although withthe development of the structured data, a series of standard which is easy to be acceptedhas been formed, the constraint of the relational data capacity makes it hard to adapt tothe massive data demand.Based on the study and analysis of distributed storage technology, combined withthe current mainstream storage related product design ideas and applicationrequirements, this paper has proposed a set of reasonable solutions for storing massivestructured data, and has implemented a structured storage system. With the distributedidea, the system will split the structured data into lots of fragmentation which will beindependently stored in different storage node, so that each node can be fully usedThis paper mainly researches on the following points:Frist, design of the structured storage system. Use the distributed idea, the systemcan be deployed to multiple nodes and the operation of the data can be divided intomultiple independent sub-processes. Therefore, the corresponding function can becompleted by each independent service node.Second, Design and implementation of the system metadata cluster. Utilizeclustering technology to store massive metadata information to meet the storage designrequirement of the whole system. Improve the network throughput of the system byusing the Epoll non-blocking technology; hence the metadata storage cluster will notbecome the bottleneck of the whole system. In the meantime, according to the feature ofthe cluster, centralized scheduling management strategies can be employed toimplement the loading, balancing and scheduling within the cluster, to make sure theaccess of the hot spots of the cluster and the balance of the storage load.Third, fault-tolerant design and implementation of the system metadata cluster.The design of the fault-tolerant solution for metadata cluster combines duplicate redundancy and hot-standby redundancy. To improve the reliability and robustness ofthe cluster, for scheduling node apply hot-standby redundancy, while for storage nodeuse duplicate redundancy.The result data of functional testing and stress testing indicates that the entirestructured storage system has complete function, can work normally under pressure, andis able to complete the operation request from large-scale users in a reasonable delaytime.