Designing High Performance and Scalable Unified Communication Runtime (UCR) for HPC and Big Data Middleware

The computation and communication requirements of modern HighPerformance Computing (HPC) and Big Data applications are steadily increasing. HPC scientific applications typically use Message Passing Interface (MPI) as the programming model, however, there is an increased focus on hybrid MPI+PGAS (Partitioned Global Address Space) models for emerging exascale systems. Big Data applications rely on middleware such as Hadoop (including MapReduce, HDFS, HBase, etc.) and Memcached. It is critical that these middleware be designed with high scalability and performance for next generation systems. In order to ensure that HPC and Big Data applications can continue to scale and leverage the capabilities and performance of emerging technologies, a high performance communication runtime is much needed. This thesis focuses on designing a high performance and scalable Unified Communication Runtime (UCR) for HPC and Big Data middleware.;In HPC domain, MPI has been the prevailing communication middleware for more than two decades. Even though it has been successful in developing regular and iterative applications, it can be very difficult to use MPI and maintain performance for irregular, data-driven applications. PGAS programming model presents an attractive alternative for designing such applications and provides higher productivity. It is widely believed that parts of applications can be redesigned using PGAS models - leading to hybrid MPI+PGAS applications, and improve performance. In order to fully leverage the performance benefits offered by the modern HPC systems, a unified communication runtime that offers the advantages of both MPI and PGAS programming models is critical. We present MVAPICH2-X - a high performance and scalable Unified Communication Runtime that supports both MPI and PGAS programming models. This thesis also targets at redesigning applications making use of hybrid programming features, for better performance. With our hybrid MPI+PGAS design using Unified Communication Runtime, the execution time of Graph500 was reduced by 13X, compared to existing MPI based design at 16,384 processes. Similarly, the sort-rate of data intensive Out-of-Core Sort application was improved by a factor of two, using our hybrid designs.;The requirements for Big Data communication middleware are similar to those used in HPC. Both these rely on low-level communication techniques. However, the performance of current generation Big Data computing platforms, such as HBase and Memcached remains low. One of the fundamental bottlenecks in delivering high-performance on these platforms is use of the traditional BSD sockets interface and two-sided (send/recv) communication. Such communication semantics prevent the use of Remote Direct Memory Access (RDMA) and associated features of modern networking and I/O technologies. This thesis focuses on the design of a lightweight, scalable and high performance communication runtime for Big Data middleware, such as HBase and Memcached. With our UCR design, Memcached operation latencies were reduced by a factor of 12X, and HBase throughput was improved by 3X times.