Compiler-driven I/O minimization

Most scientific programs have large input and output data sets that require out-of-core programming or use virtual memory management (VMM). VMM is not an effective approach because it often results in substantial performance reduction. In contrast, compiler driven I/O management will allow a program's data sets to be retrieved in parts, called blocks or tiles. Out-of-core programming defines the partitions of the data sets and the control of I/O. Comanche (COmpiler Managed CaCHE) is a compiler combined with a user level runtime system that can replace and out-perform standard VMM for out-of-care programs. In addition, this system does not require any special services from the operating system and does not require modification of the operating system kernel.; Our research aims at improving the applicability of Comanche to achieve high performance of I/O-intensive out-of-core computations. This research shows that block mapping I/O management may improve the performance of out-of-core problems by one order of magnitude or more. We have studied the relationship between data access patterns and data locality optimization and created a tool for compile time data transfer analysis. Our extension of Comanche involves the use of an out-of-core programming schema for optimizing locality in loop nested calculation to minimize I/O. The final goal of our research is to integrate the new I/O optimization techniques and data transfer analysis tool into Comanche to make it more useful and powerful.