Design And Implementation Of A Resource Scheduling Platform Supporting Multiple Scheduling Systems And Multiple Task Types

With the rapid development of distributed computing technology,distributed resource management,as the core technology of distributed computing,has also attracted more and more attention.From stand-alone servers to distributed cluster environments;from standalone Docker container technology,to distributed container orchestration tools,to distributed resource scheduling systems.Distributed resource management has become more diversified and more complex.However,the architecture design and usage methods of different resource scheduling systems are very different.Whenever the technical selection of the resource scheduling system is changed,huge learning costs need to be invested,from setting up the environment,to basic use,to a customized transformation,each step requires a lot of time and energy.In response to the above problems,this thesis designs and implements a distributed resource scheduling platform,which can use different resource scheduling systems as plug-ins to manage distributed resources and provide a set of unified interfaces for users to use.Users only need to use our resource scheduling platform to submit multiple types of tasks to use distributed resources,without worrying about the differences between different resource scheduling systems at the bottom.This platform satisfies the scalability of resource scheduling system types and the scalability of task types.Through the management of master-slave roles,the platform itself has good fault tolerance.In view of the above functional requirements,the main work of this thesis is as follows:(1)Researched and analyzed the architecture design of two mature distributed resource scheduling systems Kubernetes and Mesos,combined the advantages of the two systems,and designed the overall architecture and task scheduling process of the distributed resource scheduling platform.(2)The development history and design ideas of plug-in technology are studied,and they are integrated into the access module of the resource scheduling system of this platform,so that the platform will plug-in the resource scheduling system.By installing different resource scheduling system plug-ins,it can meet the needs of the scalability of different types of resource scheduling systems.(3)Analyzed the container technology that supports environment isolation and resource limitation,adopted the most popular Docker as the execution environment,made different executors to enable different types of tasks to be executed,and formulated the executor's process specification to make Implementations that follow this specification can be accessed,which satisfies the scalability of task types.(4)The master-slave module,resource management module and task scheduling module are designed and implemented,and the basic functions of the server of the resource scheduling platform are realized.At the same time,a set of front-end pages are provided to enable users to conveniently manage tasks.Introduced a third-party distributed file system as the data center to enable the resource scheduling platform to meet the support of large task files.Finally,for the designed distributed resource scheduling platform,we built a three-node cluster experimental environment,and performed functional and performance tests on it.The test results show that the implemented distributed resource scheduling platform has reached the expected design goals,and has good compatibility and scalability.When compared with the design of ordinary container cloud platforms,this platform can access more types of resource scheduling systems and support more types of tasks.