Design And Implementation Of Efficient Metadata Index In Distributed File System

With data grows explosively, the file system’s scale expands quickly, the number offiles in file system has reached to hundreds of millions，distributed file system such asGFS(Google File System) can handle the storage problem of massive files. Metadatadescripts the basic features of file in file system, both the system administrator and normalusers need to quickly find the required files by file metadata, how to organize and indexthe metadata in order to ensure efficient metadata access and query is a serious problem indistributed file system.In order to solve the efficient access and query problem in file system,we analyzes theaccess、query and distribution characteristics of file metadata, presents a metadataorganization solution. This solution considering with metadata query function, divides thefile system metadata into several partitions hierachically, each partition’s metadata aresequentially stored on disk, using bloom filter to build a content table for each partition forquickly narrowing the metadata search space, using multidimensional search tree(kd-tree)to create in-memory index for supporting the multidimensional attribute query. In addition,create operation log for each partition, the metadata update will be merged into index toensure real time metadata search, and periodically flushed to disk.We develop Mspy(Metadata Spy), a metadata management prototype for DistributedNetwork File System(DNFS), and evaluate Mspy’s search performance by realexperiments. The experiments show that Mspy’s performance is3-8times higher thantraditional solution，make most metadata queries can be finished in one second.