Implementation And Optimization Of Metadata Cache Consistency In Distributed File System Client

With mass information generated, the performance requirements of data storage and access become increasingly higher. In a distributed file system, metadata access is the only way to access the data. In all file system access, metadata access accounted for more than 50%, thus increasing the metadata access performance is particularly important. Metadata access performance can be improved by implementing the client's local cache. However, to ensure that multi-client metadata cache consistency is one of the key techniques. The realization of the consistency function will bring some overhead, lead to reduced performance, so we need to weigh the functionality and performance of consistency, and use some optimization techniques to guarantee the correctness and performance of the file system cache consistency.Delegation lock is implemented based on Zookeeper consistency coordination service to ensure the metadata cache consistency. In order to reduce the performance overhead caused by maintaining consistency, the system is optimized from the following three aspects: first, we analyzed request process on the Zookeeper server. According to the overhead ratio of disk and network IO during lock operation, we got 90% increase of locking performance by replacing hard disk with SSD. Second, by comparing the subtree lock method of Lustre file system, we put forward a method called directory range lock which can get all sub-file's metadata cache permissions of the directory, to reduce the network traffic overhead. And the experiment results showed that the metadata OPS and concurrent access performance were better than the non-optimized system and Lustre system; finally, in order to support metadata cluster, unique ID method is implemented to further improve metadata access performance. The results showed that in the same client scale, compared to a metadata server, the metadata OPS of multiple metadata server cluster all improved.