Research On Optimizing PNFS-Based Metadata Server By Access Hotspot

The rapid development of high-performance computing and network technology puts forward increasing request to the file system. To store and access files quickly becomes an important method to improve the performance of the overall system. Distributed file system can separate the control flow and the data flow and thus accelerates the file access efficiently. But with the continuous expansion of the system scale, the pressure on metadata server also increases. The use of metadata server cluster can increase the efficiency of metadata access and enhance the reliability of the whole system.On studying some recent metadata management and organization methods, much research is done on the existing pNFS metadata server cluster based on static subtree. It is found that the cluster can't balance the burden when there are hotspots in the system. Meanwhile, the Zipf-like distribution of file access mentioned in many papers is noticed. A novel system is designed and implemented by taking advantage of the characteristics. The system gets the reading hotspots by counting the metadata access information and distributes the hotspot metadata into an individual subtree in all machines in metadata server cluster without changing the subtree structure. Clients are able to choose any metadata server to get the hotspot metadata and this method increases the parallel service ability of the whole cluster. Because of the high frequency of accessing hotspot files, this strategy can solve the problem that major workload locates several servers caused by static subtree partition.In a metadata cluster composed of 4 machines, the improved system shows about 3 times the throughput of the old system when the directory depth is 5 and only hotspot metadata is accessed. It also upgrades 32.7% the throughput in the Saskatchewan-HTTP workload test. The load balance is also better. The test result indicates that replica access hotspot subtree can improve the system performance and balance the load efficiently.