Study On Optimization Technologies Of Distributed Hybrid Storage System For High Performance Computer

With the advent of the big data era,the amount of data handled by applications has soared,and the combination of big data applications and high-performance computing has become increasingly tight.While the performance development of high-performance computer storage systems is lagging behind.In order to improve the performance of storage system,we designed and implrment ed a distributed hybrid storage system based on DRAM and SSD.We propose a data migration algorithm based on the I/O characteristics of SSD,and we optimized the metadata service of the system.We carried out the following aspects of research work:(1)Design and implement a hybrid storage system based on DRAM and SSD,design the storage space management of hybrid storage system,and optimize the data transmission mechanism based on Infiniband RDMA.(2)Put forward a strategy of data migration based on the I/O characteristics of SSD as well as the historical access information,and optimize the read and write performance in different access modes.(3)Meta-data access optimization through file striping reduces the overhead of metadata operations during data access and significantly improves the small I / O bandwidth.(4)Extended the system metadata service module to cluster architecture,improved the system metadata service capabilities by user directory subtree metadata division strategy,while making the system better suited to the actual production environment.We have tested the design strategy and optimization of hybrid storage system in Tianhe engineering simulation system.The test results show that the data migration algorithm in hybrid storage system is effective and superior to LRU algorithm in the small I / O access mode,and the metadata service capability has a nearly linear increase.The performance of the hybrid storage system reaches the performance of a completely DRAM-based storage system 75% to 90%.