The Challenges And Solutions Of Parallel Storage Systems For I/O-intensive Applications

Along with the rapid development of high-performance computing technology, the system performance of high-performance computer is upgrading. Currently, the high-performance computing has entered PetaFLOPS era, and is embarking on its entry into Exascale era, accompanied by the magnitude of data storage exceeding Petabyte. However, the upgrading of storage system which is one of the most important parts of the high-performance computer system has far fallen behind. And the gap between the performance of storage system and that of computing has been severely limiting the utilization of the advanced system performance of high-performance computer.This paper aimes at the I/O bottleneck problem of the HPC system, studies parallel filesystem and storage system architecture under the background of the important application of Oil seismic exploration data processing. At last, we proposal some schemes to alleviate the I/O bottleneck problem we faced now and under the future Exascale computing condition.Distributed File System is the direct manager of the storage subsystem of high-performance computer, and its way of management directly influences the utilization of the storage system. Therefore, this paper intensively analyzes the Lustre File System, including its architecture, the process of data writing and reading, the process of file locking and unlocking and its cache strategy. And then find out the shortage being existed in the Lustre File System.A typical CPU-bound and I/O intensive application is the data processing of Oil seismic exploration. Hence, this paper carefully studies its I/O process and analyzes its I/O pattern as well as its I/O characteristic, in order to further explore the problem of storage system in computing application.To the problems of storage server, such as insufficient disk bandwidth output and insufficient system polymerization bandwidth, this paper proposes the method of VDISK. It uses the memory resources of computing node to store the data of the server and agents the storage server to provide the data to other computing node, leading to the stress reduction at the server side and the rise of the system polymerization bandwidth.System architecture is important to determine the performance of the storage system. Therefore, basing on the storage problems reflected in the data processing of Oil seismic exploration and from the perspective of system structure, this paper proposes the Tiered Storage System which increases Local storage resources of the computing nodes and provides data needed together with the Global storage to the computing nodes, in order to improve the storage performance and provide high performance service. In addition, three management solutions of the Tiered Storage System are designed. Finally, the protomodel of the Tiered Storage system is implemented.