Research On Client Cache Optimization Technology For Distributed File System On High Performance Computer

With the era of Big Data,high-performance applications are closely integrated with big data,and the amount of data from observations,experiments,and tests applied in scientific research fields is rapidly increasing.The scale of high-performance applications is increasing,and applications generally require high-intensity of data I/O,the I/O demand for high-performance applications is growing rapidly,demanding the high-performance computer storage subsystems to implement with high parallelism,high bandwidth,low latency,and large capacity.The storage systems are under severe I/O pressure.Cache technology is a classic technology for mitigating I/O bottlenecks in high-performance computer systems.It can alleviate bottlenecks in distributed file system client I/O performance and improve the performance of high-performance computer storage system clients when dealing with typical high-performance applications.The caching technology proposes to use the free memory of the distributed file system client node to build the client cache,and further optimize the client cache by using the high-performance computer to support the RDMA data transmission mechanism,and optimize the cache for the I/O characteristics of typical high-performance applications.Consistency strategy and metadata access mechanism.The paper mainly has the following parts:(1)Design and implement the client-side caching of distributed hybrid storage system,build client-side cache with high-performance computer node free memory resources,and optimize client I/O performance through pre-reading and write-back.(2)Utilize the high-performance computer that support the mechanism of RDMA data transmission,and manage the client cache and RDMA buffer uniformly,to avoiding multiple memory copies inside the client during data I/O,thus making efficient use of free memory.At the same time improve the execution performance of the user program.(3)Study the client-side cache consistency policy and the metadata optimization strategy.For the access characteristics of typical high-performance applications on TH-1A,implement a phase-consistent strategy to optimize the performance of the client-side cache and pass the client.The end metadata cache optimizes read performance.(4)Deploy the system to the TH-1A high-performance computer computing node,test the client I/O performance through the benchmark program and typical I/O intensive applications,and conduct in-depth analysis of the test results.The test results show that the client-side caching mechanism implemented in this paper can greatly improve the I/O performance of the client.