Compiler techniques for data parallel applications using very large multi-dimensional datasets

Processing and analyzing large volumes of data play an increasingly important role in many domains of scientific research. The goal of this work is to provide high-level language support for implementing such data-intensive applications. Our approach is based on the fact that a data-parallel framework provides a convenient interface to large multi-dimensional datasets resident on persistent storage. Allowing these applications to be programmed from a high-level environment facilitates the development process by allowing programmers to concentrate on the applications issues, leaving systems issues up to the compiler and run-time support.; In this dissertation, we designed a set of compiler analysis techniques for extracting information from data-parallel loops written in high-level object-oriented programming languages. We also designed a set of execution strategies that use the information extracted earlier to execute the loops on distributed memory machines. Our techniques targeted code generation for regular and irregular applications with dense and sparse datasets. We also developed strategies for reducing communication volume for the applications. Other techniques were also developed to contribute to the efficiency of the compiler generated code, for example, a conditional motion framework that helps alleviate some of the overhead incurred by our execution strategies.; We built a prototype compiler using Titanium, a Java dialect with data-parallel extensions, as front-end. Our compiler analyzes the parallel loops and makes extensive use of the Active Data Repository (ADR), the target run-time system. We used this prototype to evaluate the efficacy of our techniques and the performance of the compiler generated applications. We also studied further optimizations that can be applied to the compiler generated code and evaluated their performance.