Research And Design Of Cloud Workflow Service Farmework Supporting Multiple Infrastructures

Cloud computing is an emerging computing paradigm that can offer unprecedented scalability and resources on demand, and is getting more and more adoption in the science community, while scientific workflow management systems provide essential support such as management of data and task dependencies, job scheduling and execution, provenance tracking, etc., to scientific computing. As the world is entering into a “big data” era, and the data that needs to be processed generally grows faster than computational resources and their speed, traditional workflow systems are unable to manage the ever increasing data scale and analysis complexity. To address the challenges from big data, a service framework is proposed based on multiple infrastructures for cloud workflow. The framework is capable of providing workflow as a service in the Cloud, and enables the workflow to address peta-scale scientific problems.In this thesis, the basic theory of cloud workflow is discussed. Then the novel and improved framework of cloud workflow is proposed, in which scientific workflow management systems and Cloud computing platforms are integrated seamlessly. The service framework also supports workflow-as-a-service model and can provide scientific researchers and developers with a convenient and efficient cloud workflow service platform. The contribution of this thesis can be concluded as follows:1) The major challenges of running scientific workflows on the Cloud are analyzed and four possible solutions for deploying the proposed reference architecture in a Cloud computing environment are discussed. Taking all the pros and cons of every solution, together with actual requirements, into consideration, appropriate integration solution is chosen for the framework design.2) A reference service framework is proposed which covers all the major aspects involved in integrating various scientific workflow systems with various Cloud computing platforms. The service framework consists of eight major components, including Cloud workflow management service, Cloud resource manager, etc., and 6 interfaces between them. A reference framework for the implementation of Cloud Resource Manager is also presented, which is responsible for the provisioning and management of virtual resources in the Cloud.3) To verify the feasibility, performance and scalability of our cloud workflow service framework, the service framework is realized based on OpenNebula and Eucalyptus cloud platforms. The Cloud Resource Manager is implemented and deployed in the Cloud to support provision and management of cloud resources. Efficient task scheduling service is implemented based on Falkon task scheduling framework to support large-scale task dispatching. A wrapper service is encapsulated over the original Swift system to support the workflow-management-as-a-service model. A client-side development and submission tool is also provided for application specification and submission.4) Comprehensive tests are designed to verify the feasibility and performance of service framework, and the efficiency of cloud resource management. The capability of the solution is also demonstrated by using a NASA MODIS image processing workflow and a production deployment on the Science@Guoshi network with support for the Montage image mosaic workflow.