Research And Implementation Of Distributed File System Metadata Extended Management

With the rapid development of the Internet, the scale of data in various industries is increasing, vast amounts of data and documents need to be stored, which brings an enormous challenge to distributed file system. Distributed file system stores metadata and data separately. Metadata management system manages metadata and is the most important part, but current metadata management system cannot solve the challenge. Based on this situation, zone-based metadata extended management system was proposed, where zone is unit for metadata to be organized and is partition granularity of metadata. Metadata partition granularity directly affects scalability, concurrency, data consistency and load.balance of distributed file systems. After analyzing the traditional metadata partition techniques, this paper selected zones as the most appropriate partition granularity.Zone-based load balance strategy was proposed to tackle distribution problems of zones and migration problems of zones when hot spots appear in metadata server cluster. When new zones need to be allocated, the strategy randomly assigns the zones to a metadata server by hash strategy, which ensures that the distribution is uniform. When one metadata server becomes accessible hot spot, some zones will be moved to other metadata servers, which changes destination of clients and makes full use of system resources in metadata server cluster, thus achieves load balance.In addition, the paper also proposed mechanisms to protect the availability of the zone-based metadata management system. Hot standby mechanism and log-based fault recovery mechanism ensure the smooth operation of the zone-based metadata management system.Finally,’zone-based metadata extended management system was realized in the distributed file system named SandBox. Test results have proved that zone-based metadata extended management system is easy to be extended and supports high concurrency and load balance.