Scheduling Algorithms For Analytical Cloud Workflow

With the rapid development of cloud computing technology and its advantages of high reliability,high portability and pay-as-you-go,it has attracted organizations and individuals to deploy applications and services to the cloud computing platform.Many data analysis workflow applications are also gradually completed the migration to the cloud computing environment.Analytical workflow applications deployed in the cloud computing environment take advantage of flexible buying options and flexible resource capacity during scheduling and executing.New problems and challenges also emerged.These challenges are mainly in the following aspects:(1)Flexible buying options combination problem.Chosing different options may lead to a cheaper scheduling plan.(2)Store or recompute the intermediate dataset decision problem.Since spot instances would cause failure,intermediate dataset would be used in the future.The cost of store or recompute this dataset is different.Make this choice wisely can cut down the total cost.(3)Workflow with dynamic structure problem.Some workflow model structure varies from user inputs.For example,give a hyper-parameter optimization analytical workflow,there are a bunch of parallel branches generated after instantiation.Combine these branches and reduce branches to a lower size would save computing resources and save total cost.To solve the above problems and cut down user's cost during analytical workflow execution,the main research work and innovative results reflected in the following aspects1.We propose a deadline driven workflow scheduling algorithm which is able to use both spot instances and on-demand instance to reduce the cost.2.We propose a analytical workflow scheduling algorithm that selective store the inter-mediate datasets with considering the trade-off between regeneration cost and storage cost.3.Based on the NP-Complete problem,number partitioning,we propose a algorithm that deals with hyper-parameter optimization workflow.This algorithm can reduce a large scale of parallel branches to a smaller scale of balanced branches.We develop a simulator to run experiments with the above algorithms,results show the effectiveness in saving money and ensuring the completion time.