Dynamic Optimization Of Scientific Workflows In Cloud Environment

The scientific experiments based on large scale computing resources and mass storage devices have become an important means of scientific exploration,engineering design and verification.Scientific workflows is a new type of application,which can support the automatic arrangement,execution,monitoring and tracking of scientific experiments,distributed collaboration,resource sharing and improve the automation of scientific experiment.Compared with the common workflows based on business logic,scientific workflows is characterized by both computation intensive and data intensive.Traditional computing environment has been difficult to meet the requirements of scientific workflows for computing resources.Cloud environment can be used as an ideal environment for scientific work because of its unlimited computing and storage resources.For an experimental organization,its scientific workflows engine needs to support the execution of a large number of scientific workflows,and the scientific workflow execution engine needs to rent computing and storage resources from cloud operators.Because of the characteristics of both computational intensive and data intensive,data distributed parallel tasks need to be executed on the cluster.Renting cloud resources need to pay a certain fee to cloud operators.Due to the dynamic nature of scientific workflows execution,it is necessary to know the load situation of the cluster,and to achieve the scalability of the resource.In addition,the resource usage of each node in the cluster is different.The tasks should be allocated according to the resource situation of each node to improve the balance of the cluster load,so as to improve the efficiency of the implementation of the workflow and save cost.The balanced adjustment of the cluster load can be based on the current computing resources and the execution status of the workflow,and has the characteristics of typical Markov chain,this paper establishes the Markov prediction model,the main purpose of the model is to predict the next time on the cluster load state for dynamic expansion of cluster resources.In addition,a cluster may deal with different types of tasks at the same time,and different types of tasks have different demands on resources.On this basis,this paper proposes a load adaptive task scheduling strategy.The strategy selects the node load capacity assessment model according to the load type,so that the cluster node can accurately measure the load bearing capacity of each node under the unified evaluation standard,so as to more rationally assign the task.The experimental results show that the strategy can make the cluster more rationally allocate tasks to the nodes on the basis of the appropriate load,so that the cluster reaches the state of load balancing,further improve the utilization of cluster resources,the efficiency of the operation and reduce operating cost.