| High-performance scientific computing in a modern age uses parallel techniques at a scale of hundreds of thousands of processors. These large-scale applications have I/O system workloads that are primarily driven by small, sparse I/O operations. While parallel file systems have provided application developers with scalable peak I/O bandwidth for large, contiguous I/O operations, the noncontiguous I/O access patterns common to scientific applications have remained a serious performance problem. Our work in this area has contributed several solutions toward improving the gap in I/O performance and other system components through new list I/O and datatype I/O methods. Detailed studies on basic I/O characteristics and application-level I/O strategies have led to performance-oriented suggestions for application programmers on how to best use these new I/O methods. In this paper, we also discuss our optimized DLM and versioning approaches for implementing atomic noncontiguous I/O operations that are the building blocks for handling challenging problems in redundant storage, consistent distributed data layouts for parallel processing, and other producer-consumer problems. |
