Open HPC++: An open programming environment for high-performance distributed applications

With the recent advances in computer network technology, it is becoming easier to connect geographically distributed resources such as massive supercomputers, advanced display devices and large databases. Such a distributed collection of supercomputing resources is called a metacomputer and the high-performance distributed applications running on a metacomputer are called metaapplications. Development of such metaapplications can be extremely challenging because of their complex runtime environment coupled with their requirements of high-performance and Quality of Service (QoS). Such applications typically run on a set of heterogeneous machines with dynamically varying loads, connected by heterogeneous networks possibly supporting a wide variety of communication protocols. In spite of the size and complexity of such applications, they must provide the high-performance and QoS mandated by their users. In order to achieve the goal of high-performance, they need to adaptively utilize their computational and communication resources. Apart from the requirements of adaptive resource utilization, such applications have a third kind of requirement related to remote access QoS. Different clients, although accessing a single server resource, may have differing QoS requirements from their remote connections. For example, one client might want to use secure communication with mutual authentication and data encryption; while another client may only need guaranteed data delivery. A single server resource may even need to provide different QoS for different clients, depending on various issues such as the amount of trust between the server and a given client. These QoS requirements can be encapsulated under the abstraction of remote access capabilities. Metaapplications need to address all the above three requirements in order to achieve the goal of high-performance and satisfy user expectations of QoS.; This thesis presents Open HPC++, a programming environment for high-performance, applications running in a complex and heterogeneous run-time environment. Open HPC++ provides application level tools and mechanisms to satisfy application requirements of adaptive resource utilization and remote access capabilities. Open HPC++ is designed on the lines of CORBA and uses an Object Request Broker (ORB) to support seamless communication between distributed application components. In order to provide adaptive utilization of communication resources, it uses the principle of Open Implementation to open up the communication mechanisms of its ORB. By virtue of its open architecture, the ORB supports multiple, possibly custom, communication protocols, along with automatic and user controlled protocol selection at run-time. An extension of the same mechanism is used to support the concept of remote access capabilities. In order to support adaptive utilization of computational resources, Open HPC++ also provides a flexible yet powerful set of load-balancing mechanisms that can be used to implement custom load-balancing strategies. We will discuss the performance evaluation of the Open HPC++ adaptivity and load-balancing mechanisms. We will also present a set of applications to illustrate how these mechanisms may be used to satisfy the three requirements.