Data-intensive Workflow Scheduling Based On Stage Division In Cloud Environment

The cloud computing,an emerging computing model,provides a good operating environment for data-intensive workflows.The operation of data-intensive workflows often requires the computing resources of multiple cloud service providers to support,and reasonable task scheduling is the key to its efficient operation.For a typical data-intensive workflow,there are many tasks and a large amount of data transmitted between them,involving many cloud service providers.The double optimization of the overall completion time and overall execution cost of the workflow operation has always been a hot issue for current data-intensive workflow scheduling.For this reason,this thesis proposes a data intensive workflow scheduling method based on stage division in cloud environment.Firstly,the workflow is divided into several stages based on the data dependency of tasks,and then the tasks are scheduled for each stage through two core steps: task pre allocation and scheduling adjustment.Among them,task pre-allocation schedules multiple tasks in the current stage in parallel according to the estimated task execution time and the data transmission time between tasks to reduce the completion time as much as possible,and scheduling adjustments divide some parallel tasks based on specific conditions adjust to the same service provider for serial execution to reduce the amount of transmission between tasks without increasing the completion time of this stage,thereby reducing transmission costs and further reducing the completion time.Eventually,the dual optimization of workflow completion time and execution cost can be achieved through stage-by-stage pre-allocation and scheduling adjustment optimization.In this thesis,the feasibility of the workflow scheduling method is verified by a specific example,At the same time,a simulation experiment is designed and implemented,which shows the optimization effect of this method on the overall completion time and execution cost compared with other algorithms,and verifies the effectiveness of this method.