The Optimization Of High Performance Computer Storage System For I/O-intensive Applications

As one of the most important technologies for scientific research and exploration,scientific computing poses a great demand for the computability and data-processingability of the high performance computer, which serves as the great impetus for thedevelopment of the high performance computer system. Currently, the highperformance computation is evolving from PetaFLOPS to Exascale and meanwhile,data storage is steping into Petabyte era. High performance storage system, whoseperformance has great influences upon the integral performance of the whole system, isan important component of the high performance computer. As the wide spread of thehigh performance computing application, the problems emerging from the storagesystem of the high performance computer while dealing with data-intensive applicationshave pose a challenge to the further improvement of the application performances.In order to curb the I/O bottleneck of the high performance computing system, thispaper focused on the trace gathering of the petroleum exploration data processing, byanalysing I/O features of the the program of trace gathering and storage system’scharacteristics of the high performance computer, proposes a solution to the I/Ointensive applications. The work included is as follow:First of all, this paper summaried the basic structure, technical characteristics,potential problems and solutions. On the base of studying the problems andoptimization methods of the existing storage systems, we compared the technicalcharacteristics and technical defects, and further proposed our research program.In addition, this paper exemplified the trace gathering process of the petroleumexploration data processing, which is typical for the I/O intensive applications for highperformance computer, explored its I/O flow for data processing and analysed its I/Omode and I/O feature, uncovered the existing problems of the trace gathering.Moreover, in terms of the problems existing in trace gathering, such as frequentaccess to a small data disk, lacking of data sharing among jobs and the low efficiency ofduplicate data usage, this paper proposed a memory based method optimized for tracegathering. By storing application data in the idle memories of the computing nodes, thememory saved the disk storage system in providing data to the computing nodes, thuslower the I/O access to the disk. And this method was proved to effectively improve theperformance and mitigate the storing pressure for the disk storage system boththeoretically and experimentally.Last but not least, in views of the problems exiting in the I/O intensive applicationrepresented by trace gather, this paper, in the perspective of storage system architecture,studied improvement of the integral performance of I/O system and constructed a memory-based I/O storage system(Storage In Memory, SIM). The system, whichvisualized the local memory resources of the computing nodes and was above the diskstorage system, was expected to break the I/O bottleneck, improve the operatingefficiency of the I/O intensive application and the performance of the high performancecomputer storage system.