Big Data Platform Job Scheduling Algorithm And Simulation Implementation Based On Deep Reinforcement Learning

With the advancement and development of the times technology,the personal host,high-performance mainframe or other multi-virtual machine distributed business clusters used by people are developing at a high speed.High-level hardware and software functions have created tremendous opportunities for our work and life..But at the same time,the magnitude of the information we generate from using resources is exploding,such as daily forecasts,hourly or minute weather conditions,and our spring train ticket bookings.The easiest and most straightforward way to allow a large amount of resource allocation is to increase or increase the amount of resources,but it is associated with huge costs.It is especially important to use good resource strategies and algorithms.This paper aims to study a good general-purpose big data platform job scheduling algorithm.At the same time,the goal is to study on a multi-resource pool platform and multi-virtual machine cluster.This paper mainly chooses to train and test on the custom implementation simulation system,and makes this paper meet the requirements of real environment.The experiment has good portability,can be applied to each big data platform center according to the corresponding modification,and it also uses various visualization tools to clearly show the experimental results.The main contents of this thesis are as follows: Firstly,a job scheduling algorithm based on deep reinforcement learning is implemented.This algorithm can well cope with the online and offline job scheduling requirements of a multi-resource pool platform,and has good effects,and then multiple resources.The pool platform was promoted and the idea of ? ? reinforcement learning was introduced.Finally,the whole job scheduling simulation system is introduced in detail,and multiple modules such as job generation,job execution,and algorithm call are implemented.This thesis uses the deep learning method and combined with reinforcement learning to study the field of job scheduling.In the case of using some mature open source frameworks,the network structure suitable for deep reinforcement learning is customized and adjusted,and the whole operation process is carried out.Control and modification.At the same time,this thesis designs and independently implements the simulation system and scheduling algorithm of the whole big data platform jobscheduling,and also extends and implements in the more complex multi-virtual machine multi-resource environment.Compared with the existing algorithms,the processing speed is increased on average or the average waiting time is reduced by about 30%.It has laid a good foundation for the transplantation of real big data platforms in the future.