The Design And Implementation Of A Metadata Server Of Distributed Filesystem

With the development of information technology, the global data are increasingly expanding. In face of the storage demand for countless data, the traditional storage system has bottlenecks in scalability and performance. Distributive storage has been widely recognized in the industry due to its strong expansibility, high cost-effectiveness and good fault tolerance. More than 50% file operations in the storage systems are metadata operations. In the scenarios of read-more & write-less of massive small files with namespaces, the performance of many existing file systems is low poor due to the bottleneck of the performance and scalability of metadata service.In this paper, a kind of metadata server for cloud storage system, is proposed to support massive small files storage with high performance, high concurrency, and high scalability. The clustering design largely improves the scalability of the system. We introduced a new dynamic load-balancing strategy to reduce load thrashing and improve the precision of rebalancing among the clusters. Besides, we proposed the concept of the channel, which was an independent data storage pipe from software to hardware formed by binding the independent CPU-core to the independent physical disk. And then, a multi-channel fast key-value storage engine provides the extremely efficient performance for the underlying storage service of our metadata server, which takes full advantages of multi-core processors and parallel disks I/O.Performance measurements under a variety of benchmarks show that the metadata server can support the massive small files storage scene and has many times of performance improvement over other metadata storage systems.