Research Of A Parallel I/O Optimization Strategy For Highly Concurrent Data Access

Nowadays HPC scientific applications have become more and more data-intensive, and I/O performance is considered as a critical bottleneck in today's typical parallel I/O applications, especially when different computing processes need to access a single-shared file. In order to handle those access patterns efficiently, the MPI-IO that includes collective I/O optimization is commonly used to improve the overall parallel I/O performance. At the same time, many scientific applications currently run on parallel distributed file systems like Lustre in order to satisfy the big data storage and effective data access, which bring in new challenges for parallel I/O optimizations.In this paper, we provide a novel parallel I/O optimization system IBCS to address this problem. IBCS takes both the I/O access pattern of applications and the physical data layout information into consideration. By reorganizing the I/O requests, our proposed design makes a “one-to-one” match between I/O aggregators and I/O nodes, so that it can alleviate the access conflicts and contentions. Our proposed design first gather and analyze application's access patterns, then transform them into physical data layout information. Our optimization strategy also implements a new aggregator choosing function and file domain partitioning method to balance the data load and align lock boundary. Finally it reorganizes I/O requests to guarantee that no contention occurs between processes during each iteration where each sub-request divided from a large one being delivered to the underlying parallel file systems. Our proposed strategy is based on iterations, which provides a finer-grained parallelism and more flexibility for reorganization in a collective I/O operation. It thus can reduce the access contentions and improve the I/O performance without introducing extra communication cost.The evaluation results based on Lustre file system and MPI-IO access pattern shows that with different sizes of intermediate memory, numbers of OSTs and computing processes, our proposed strategy has a promising performance improvement by more than 30% compared with the original implementation for parallel I/O applications.