Research And Implementation Of Efficient Task Scheduling Technology In Distributed Computing System

With the development of big data,artificial intelligence,the demand for task scheduling is increasing,the application scenarios are constantly enriched,and the triggering conditions of tasks are becoming more and more complex.The existing distributed task scheduling system is difficult to deal with complex business scenarios.On the one hand,the existing scheduling systems that support DAG task scheduling and batch task scheduling are mostly for Map Reduce tasks and lack support for tasks on other platforms,on the other hand,the existing distributed task scheduling systems lack support for efficient task scheduling algorithms,task scheduling is inefficient,and uneven load is prone to occur.In view of the above problems,this thesis establishes a distributed task scheduling system model,proposes task scheduling algorithms for DAG tasks and batch tasks,designs and implements a distributed task scheduling system that supports multiple task types.The main work and contributions of this thesis are as follows:(1)Aiming at the problem of DAG task scheduling,a task scheduling failure model is established,and a DAG task scheduling algorithm GA-EFT is proposed.The GA-EFT algorithm narrows the search range to the effective solution space,thereby improving the problem of slow convergence when the genetic algorithm solves the task scheduling problem.Simulation experiments show that the GA-EFT algorithm converges quickly,has good scheduling performance,and can effectively reduce the time wasted due to task failure.Compared with traditional genetic algorithm,ant colony algorithm and HEFT algorithm,the average failure time ratio decrease by 12.8%,16.4% and 30.5% respectively.(2)Aiming at the problem that the heuristic and meta-heuristic DAG task scheduling algorithm takes a long time to process DAG tasks with a small optimization space,an adaptive DAG task scheduling algorithm ORPBA is proposed.The ORPBA algorithm predicts the optimal space according to the basic characteristics of the DAG task,and selects an appropriate task scheduling algorithm to schedule it.Simulation experiments show that the ORPBA algorithm reduces the total time spent on task scheduling and execution by 11.9% compared with the GA-EFT algorithm.(3)Aiming at the problem of batch task scheduling,the SAMM algorithm is proposed.The SAMM algorithm improves the performance of batch task scheduling on the basis of ensuring the load balancing of the execution nodes by adaptively adjusting the load information collection scheme.Simulation experiments show that the SAMM algorithm,as an improved scheme of the Max-Min algorithm,takes 21.6% of the average time of dynamic Max-Min algorithm.Compared with the static Max-Min algorithm,the average scheduling performance is improved by 19.6%.Compared with the dynamic and static algorithms,the total time spent on the batch tasks has been significantly reduced.(4)Design and implement a distributed task scheduling prototype system for business operations that supports DAG tasks and batch task scheduling,and conducted functional and performance tests.The test results show that the prototype system has achieved the expected design goals.