| In an era of data explosion and analysis, researchers across the globe are trying to convert massive quantities of complex data into useful knowledge using Computational and Data-Enabled Science and Engineering (CDS&E) applications. CDS&E applications are gaining traction as an important dimension of Science, Technology, Engineering, and Mathematics research. These applications require powerful processors with fast interconnects, extreme large scale, and elastic resources. While high-end High Performance Computing (HPC) resources provide the necessary requirements, the complexity of such systems has grown exponentially. Furthermore, due to their high cost, limited availability, and high demand, the queue wait times to run applications on these systems is in the order of months. All of the above challenges prevent their adoption as a mainstream solution.;On the other hand, Cloud computing is emerging as a dominant computing paradigm that offers many advantages. Consequently, early adopters have looked into using Clouds to solve the HPC model challenges. Initially, CDS&E applications were run on commodity Clouds, but this was found to be appropriate only for certain classes of applications. Other approaches explored complementing HPC resources with Clouds but failed to address all challenges in the HPC environment. Cloud providers also tried to provide HPC as a Cloud using small HPC clusters connected to form a larger Cloud but were hindered by small scale and limited performance. These approaches fall short of providing the high performance necessary for CDS&E applications.;In this document, we propose a new approach to achieve the notion of HPC as a Service. This approach targets existing high-end HPC resources and investigates how a Cloud abstraction can be applied to provide a simple interface and support real-world applications. In particular, the application of Clouds to supercomputers is discussed, tested, and validated on an IBM Blue Gene/P supercomputer. The proposed framework transforms Blue Gene/P into an elastic cloud by bridging multiple systems to create HPC federated Clouds, supporting dynamic provisioning and efficient utilization, and maximizing ease-of-use through an as a Service abstraction. In order to effectively illustrate the benefits of such a concept, the proposed framework is demonstrated using a real-world ensemble application. |
