| In high-performance computing (HPC) environments, numerous factors conspire to make efficiently accessing large-scale scientific data difficult: the continually growing imbalance between compute and I/O capabilities, the data intensive nature of current and future scientific simulations, the distribution of data among many discrete storage locations in parallel filesystems, and the complexity of data access patterns in I/O workloads.;In this thesis, we propose complementary approaches to reduce both the size and complexity of data accesses to parallel storage, focusing on post-data-generation analysis workloads (i.e., I/O read optimization) across the I/O software stack. Additionally, we develop methodologies to efficiently process noncontiguous data, a common occurrence in HPC data workloads, informed by current architectural trends (e.g., GPUs). For data reduction, we explore techniques of level-of-detail analysis, which aims to reduce read costs for data analysis at the cost of reduced analysis precision. While existing level-of-detail methods, such as hierarchical Z-order sampling and wavelet multiresolution analysis, have proven useful in a number of analysis tasks, they do not provide both hard bounds on data precision and full-context views of the data, both of which are essential for a robust level-of-detail methodology.;Based on these limitations, we present a precision-based level-of-detail methodology (APLOD) for scientific oating-point data, which utilizes the floating-point format to provide well-defined I/O-accuracy tradeoffs. Data layout complexity is reduced by a deterministic partitioning of data, with low computational overhead and bounded per-point errors.;The data processing required to implement APLOD induces noncontiguous access patterns, which are ubiquitous in scientific computing, commonly seen in array subvolume accesses, spatio-temporal accesses, etc. Processing these accesses, specifically for I/O, have been explored at various levels of the I/O software stack. For APLOD in particular, we have shown scalable integration with the ADIOS high-level I/O library.;At the middleware level, specifically MPI and MPI-IO, we show the ability to map the APLOD format and data transformations onto an MPI datatypes representation, which enables the MPI runtime to efficiently communicate noncontiguous data without user intervention. One significant use-case missing from such noncontiguous data processing is the ability to efficiently process data residing in graphics processing unit (GPU) memory. Hence, we develop an MPI datatypes processing algorithm optimized for GPU-resident data, utilizing the massively parallel nature of GPUs. We demonstrate low processing overhead for both regularly-structured data and irregular data, compared to methods based on PCIe direct memory access (DMA).;Finally, reducing the magnitude of accesses is but one piece of the puzzle--- complex, noncontiguous access patterns are exceptionally difficult to process effectively, especially at scale. With this problem in mind and spurred on by recent developments and optimization opportunities in HPC storage systems, we explore optimizations for complex access patterns, such as those exhibited by APLOD, in the space between middleware I/O drivers (e.g., MPI) and objectbased storage systems (e.g., PVFS). We do this by exploiting direct object-storage semantics to dynamically create partially replicated data optimized for differing access patterns, as well as by leveraging integrated I/O tracing and analysis to make intelligent decisions to drive our replica-based optimizations. Our method is shown to be effective at improving I/O performance for several common noncontiguous access workloads. (Abstract shortened by UMI.). |
