Study On Key Technologies Of Distributed Memory File System For High Performance Computers

With the development of HPC technology and HPC application technology,high-performance computers have been applied more widely.Simultaneously,the I/O intensity of some typical HPC applications is getting higher with larger need in storage.Aiming at the severe I/O bottleneck of the storage system when dealing with such I/O-intensive applications,this paper presents designing and implementing a distributed memory file system based on the architecture chracteristics and available resources of high performance computers.To achieve this goal,following researches including 5 aspects are carried out:(1)Study the main technologies of distributed file system in metadata management,data distribution,consistency and availability.Analyze some typical distributed file systems and related works in the field of HPC.(2)Design and implement a distributed memory file system,which is reffered as DMFS,for high-performance computers.Implement metadata service of high availability,dataserver groups supporting dynamic expansion,object-based memory block and RDMA-based data transfer mechanism.(3)Study the replica management strategy suitable for DMFS.Implement a replica placement and selection strategy based on the combination of access features of users and comprehensive evaluation indexes of dataservers,which can balance the file access revenue and the replica creation cost.(4)Design and implement a convenient and efficient I/O trace tool based on DMFS,which can track the I/O process of applications running on DMFS and provide effective technical means for analyzing the I/O characteristics of such applications.(5)Test the I/O performance and verify the practical value of DMFS through benchmarks and I/O-intensive applications on high performance computer TH-1A.