Supporting data intensive applications in a heterogeneous environment

As high performance networks become more pervasive, there is increasing interest in making collective use of computational resources. Recent work in this area has converged to the notion of a Computational Grid, that attempts to present a large collection of distributed resources in a uniform way that is easy to use. Grid research covers many areas including low level infrastructure work such as network quality of service, service level work such as authentication and fault tolerance, up to higher level work on how applications should interact with the grid. However, providing support for efficient exploration and processing of very large datasets stored in distributed storage systems remains a challenging research issue.; In this dissertation, we investigate how such data-intensive applications can operate efficiently in a heterogeneous environment, such as the computational grid. We designed the filter-stream programming framework that can be used for restructuring data-intensive applications into a set of processing components, referred to as filters. This framework explicitly supports distributed execution and is designed to optimize resource usage. We also developed the DataCutter runtime system for efficiently executing filter based applications and use this runtime system to experimentally evaluate runtime behavior. The flexibility resulting from the component structure, combined with the overall structure and component characteristics exposed by the filter-stream programming framework, enable several performance optimizations. We quantify possible optimizations manually through several detailed case studies. The choice of which host a filter executes on, or controlling parallelism introduced by multiple concurrent filters, are examples of the types of performance optimizations studied. Finally, we present an automated scheduling algorithm to approximate the benefits of the manual optimizations.