Optimal Energy--efficient And Balancing Scheduling Algorithms For Tasks Considering Completion Time In Cloud Data Center

Cloud computing is a new network technology.Broadly speaking,cloud computing is a service related to information technology,software and the Internet.Benefit from the virtualization,Cloud data centers are accelerating to become a new IT resource supply mode.The dynamics and complexity of the cloud environment put forward requirements for task scheduling strategies,not only to ensure the quality of service,but also to achieve low energy consumption in the cloud data center.The task scheduling problem of energy saving and load balancing is generally NP complete.In view of its NP complexity,further exploration is still needed.The dissertation focuses on and studies the task completion time scheduling problem of cloud data centers,and conducts in-depth research on total energy consumption optimization and load balancing in Spark,a specific scenarios.The main research work and achievements of this dissertation are as follows:(1)The resource energy-saving scheduling problem of Cloud data center is stud-ied.This dissertation focuses on CPU-intensive tasks and the purpose is to perform non-preemptive scheduling of virtual machine requests while meeting the limits of the total physical machine capacity and running time interval,so as to minimize the total energy consumption of all physical machines(Minimize total energy,Min TE).In terms of this issue,this dissertation first studies the energy consumption model of the Cloud data center.Aiming at the special case of divisible capacity configuration(SDC),the optimal solution(lower limit)of the total energy consumption of the physical machine under this condi-tion is theoretically analyzed.Then,for the general situation of indivisible task capacity configuration,an adaptive energy-saving scheduling algorithm SAVE is proposed.This method is based only on local information and uses probability functions to make deci-sions on the allocation and migration of virtual machines and to ensure the maximization of server utilization.Finally,SAVE algorithm is applied to simulation and real environ-ment and compared with DRS and eco Cloud,which are the two advanced scheduling algorithms in the industry.The test results show that the SAVE algorithm achieves sig-nificant energy saving with an average energy saving of 29.32% and 17.76% compared with those two algorithms under the simulation environment,saves 1.53% energy than the DRS algorithm in the real scene as well.(2)The optimal scheduling problem of task completion time in Cloud data center is studied.This dissertation proposes a method based on deep reinforcement learning called Deep RM?Online,which sets different reward functions to effectively solve Cloud resource management problems with different scheduling goals.This method first visu-alizes the resource usage status of the data center,and uses convolutional neural network to obtain the resource management model.Then according to the preset expert strategy,Deep RM?Online adopts imitation learning to reduce the exploration steps of reinforce-ment learning to shorten the training time of the optimal strategy.Finally,the reinforce-ment learning process is carried out,different reward functions are set according to dif-ferent scheduling objectives,and the policy gradient algorithm(Policy Gradient,PG)of reinforcement learning is used to obtain the optimization policy.This dissertation sets two scheduling goals to compare the algorithm with several heuristic algorithms and a deep reinforcement learning method.Experimental results show that the two deep reinforce-ment learning algorithms are better than the heuristic algorithms on the two scheduling objectives.Moreover,Deep RM?Online improves the convergence speed by 37.5%,and reduces the slowdown and average cycling time of tasks by 51.85% and 11.51% respec-tively compared with Deep RM,a well-known algorithm in this field.(3)The optimization of the makespan of tasks on Spark platform is studied.Spark computing engine solves the performance loss introduced by the traditional Map Reduce programming framework in iterative calculations due to frequent reading and writing of the output of the Map task that resides on the disk,the data skew phenomenon caused by the uneven distribution of source data and the uneven partitioning method of the built-in partitioning algorithm is still very prominent in Spark.To this end,in terms of the data skew problem of the Spark platform in the reduce phase,this dissertation proposes an algorithm called Resplit Reduce to evenly divide and distribute data.The algorithm first applies the cluster sampling algorithm to the intermediate data output by the Map task to estimate its key value distribution.Secondly,Resplit Reduce improves the default parti-tion function of Spark so that data can be evenly divided into multiple partitions,which promotes load balancing in the Reduce phase,so as to make fuller use of cluster resources.Besides,the algorithm also considers the heterogeneous situation of the cluster.According to the difference in computing power between each executor,Resplit Reduce uses greedy strategy to assign each task to the executor with the highest performance factor.Finally,on Spark independent heterogeneous cluster,the Resplit Reduce and several baseline al-gorithms are compared and analyzed by Word Count,Sort and Pagerank benchmark tests.Experimental results show that the algorithm proposed in this dissertation reduces the total completion time by 47.76%,32.13% and 14.47% on average on three benchmark test sets,and improves the average resource utilization of clusters by 19.67%,30.5% and 37.03%on average.