Research And Implementation Of I/O Optimization Based On The I/O Forwarding Framework

With the expansion of space for human activities, the enrichment of experimental means, the advancement of measuring instruments, many fields generate the massive data, the wide application of network exacerbate the data expansion rate. In the face of growing massive data, data processing and storage becomes a very delicate problem. In the process of research of the massive data, the required for processing data at dozens TB or even hundreds of TB, general computer clusters obviously can not meet the demand, we need to take advantage of massively parallel computing system to deal with the increasing volumes of the massive data.Today’s supercomputers rapidly developed, but between the computing power and storage capacity of the gap failed to get effective solution. The first is I/O bandwidth problem, with the growth of the single core CPU processing ability and the increasing number of CPU, in high performance computing system, continue to expand the gap between the slow growth of I/O bandwidth and the high processing ability of CPU, I/O bandwidth is more and more significant to become the performance bottleneck for high performance computing system. The second is the scalability problem, high performance computer generally increased through the computing resource scale to achieve performance improvements, this rule does not apply to the storage system, distributed file system can support a limited number of clients, therefore the scale can not be extended without limit.Currently the I/O Forwarding framework is widely used by massively parallel computing system, to solve the scalability problem of high performance computing, for example, IBM Blue Gene/P super computer and Cray XT system. In this paper, research on I/O optimization based on the typical I/O Forwarding framework, to accelerate the processing, the transmission, and the storage of the massive data in high performance computer.First of all, In order to fully overlap computation and I/O process, on the I/O Forwarding nodes we design and implement a heterogeneous buffer based on memory and SSD, the SSD as a secondary buffer, to solve the contradiction in the condition of a large amount of data input and output on the I/O Forwarding node that buffer resources are insufficient, through targeted write-back and read-ahead strategy, to optimizate the cache on the I/O Forwarding node, to reduce I/O access path and hide the I/O delay.Afterwards, in order to make full use of the spare computing capacity on the I/O Forwarding node,this paper prensent a optimization method for active data processing used in I/O path, design a active data processing service framework on the I/O Forwarding node, used to support the processing operations including data encryption、data compression、 the character statistics、 the conversion between data and table, take full advantage of the multi-core computing capacity of CPU on the I/O Forwarding node, to reduce the data storage and movement cost.Finally, implement an instance of active data processing based on the I/O Forwarding node- data compression and decompression service, implement the design and test of the parallel compression and decompression of I/O architecture in Tianhe platform, the experiment shows: the I/O optimizations can accelerate the massive data transmission both in the network and disk.