Research On Multi-objective Workflow Scheduling With Deep-Q-network-based Multi-agent Reinforcement Learning

Cloud Computing provides an effective and cost-effective distributed computing platform for executing large-scale and complex scientific and economic problems with its flexible resource provisioning model and pay-as-you-go model.Generally,large-scale scientific and economic issues are modeled as workflows,and the ever-growth data and computational demands of these applications have led to widespread research on how to efficiently schedule and deploy them on cloud environments.From the user's perspective,it is important to consider the two Quality-of-Service metrics,i.e.,the maximum completion time and total cost,basically,they want workflow applications can be completed as quickly as possible while expecting to reduce the total cost.Nevertheless,various challenges,especially its optimal scheduling for multiple conflicting objectives and diverse different workflow applications,are yet to be addressed properly.Existing multi-objective workflow scheduling approaches are still limited in many ways,e.g.,encoding is restricted by prior or posteriori experts' knowledge when handling dynamic multi-objective scheduling problems,which strongly influences the performance of scheduling.In view of the above problems,we study the research on multi-objective workflow scheduling with Deep-Q-Network-based multi-agent reinforcement learning that optimizes the make-span and total cost without requiring enormous experts' knowledge and human intervention.In this paper,we first formulate the problem with two optimization goals,i.e.,minimizing make-span and total cost.To optimize multi-workflow completion time and user's cost,we consider a Markov game model which takes the number of workflow applications and heterogeneous virtual machines as state input and the maximum completion time and cost as rewards.Then a suitable selection mechanism and reward functions are designed respectively,which can guarantee the game model to converge to a unique correlated equilibrium.Then we apply a Deep-Q-Network model in a multi-agent reinforcement learning setting to solve the Markov game model,aims to guide the scheduling of multi-workflows over Infrastructure-as-a-Service clouds.The method is capable of seeking for correlated equilibrium by two agents' collaboration and learning from environment interaction.To validate our proposed approach,we conduct extensive case studies based on multiple well-known scientific workflow templates and Amazon EC2 cloud,and make compare with traditional algorithms,e.g.,multi-objective particle swarm optimization,non-dominated sorting genetic algorithm-II,and game-theoretic-based greedy algorithm.Experimental results clearly suggest that our proposed approach outperforms traditional ones in terms of optimality of scheduling plans generated.The advantage of our proposed method exceeds 53.4% at the lowest level of make-span,and the highest level of total cost does not exceed 9.9% in terms of the percentage of difference between baseline algorithms and our proposed method.