The Design And Implementation Of High Performance Metadata Service In Distributed File System

As the rapid development of cloud computing services, there are more requirements on the cloud storage which has been one of the core technologies supporting cloud computing. Meanwhile, cloud storage itself is becoming a new service providing an easy way for users to access their data. The rapid growth of data makes the cloud storage platform face a big challenge for the system capacity and reliability. So distributed file system has been chosen as the backend of cloud storage. Metadata management is the main responsibility of a distributed file system and also very important for the system performance and stability. Because of the high degree of interdependence of metadata, it’s very challenging to design a scalable metadata service.This paper introduces the design and implementation of a distributed file system CSTORE and its metadata service. The metadata service is hosted by a cluster, an addressing strategy based on bucket and hash is applied to reach the load balance and make the service more scalable. Metadata is organized as K/V records and stored on the disk. Because of a good design on the storage engine and the key of metadata record, associated metadata could be stored continuously on disk which makes the performance a big im provement. A log-based semi-sync replication protocol is designed to solve the consistency problem caused by multi-copy of metadata. This replication strategy improves system reliability a lot with limited impact on performance. Several instances of the same user could access user’s data simultaneously from different terminals on different platforms. All the instances of a user could keep synchronous based on the operation logs. Failure detection and automatic recovery are also implemented to make metadata service more reliable and easy to manage.Finally, features of the metadata service are verified by system test. The performance of metadata service is also tested and the metadata service of CSTORE has an excellent performance when compared with Ceph.