| High performance jobs in the cloud are typically performed in containers using distributed computing.Existing schedulers in the cloud environment such as Kubernetes and Docker Swarm have a single resource scheduling method for high performance jobs.In complex scenarios,there are still some problems such as cluster resource utilization and job execution efficiency being reduced due to ignoring the dependencies between containers and the topology between GPUs,and unreasonable allocation of idle resources in multi-tenant scenarios.To solve these three problems,this thesis studies three different high performance job scheduling methods in cloud environment,and develops a corresponding scheduler based on Kubernetes technology.The main work includes:1.A coarse-grained scheduling method based on container control dependency is presented.This method first analyzes the control dependencies among containers,then prioritizes the containers within a group according to the dependencies,and then coarse-grained dispatches the containers containing control dependencies as a unit,taking into account the resource consumption of an entire group of containers during the dispatch process.This method solves the problem of job processes waiting for each other without considering the dependency between containers,and improves the efficient utilization of resources in the cluster.2.Design and implement a multi-tenant scheduling method which combines dynamic allocation of resources with preemption.This method uses a water filling model to dynamically calculate the idle resources of each tenant.Under the situation that the cluster is fully resourced as a whole,allocate idle resources dynamically to improve resource utilization.In the case of resource shortage due to new jobs created by the tenant,the inter-tenant and cross-tenant preemption should be carried out according to the resource quota and priority of the tenant to meet the resource fairness of the tenant.This combined scheduling method can improve the utilization of idle resources in the cluster and the unfair use of resources caused by purely preemptive scheduling.3.A GPU topology-aware scheduling method is presented.This method builds the GPU topology of the working node,records the communication bandwidth between different GPU combinations according to the connection type,senses the GPU resource nodes and topology according to the actual needs of the job during the scheduling process,and selects the topology combination with the best communication bandwidth for scheduling.This method can reduce the communication overhead between jobs and improve the efficiency of jobs.4.The above scheduling method is implemented based on Kubernetes.In order to adapt the scheduling method in this paper,the dispatch unit status model and multi-layer dispatch queue are improved on the existing Kubernetes dispatch module,and the monitoring and alarm module,log module and disaster tolerance module are developed.The monitoring module mainly records the performance indicators such as the running status of the process inside the dispatcher and the time-consuming of dispatching jobs,and gives timely warning when an exception occurs.The log module is responsible for uniform logging of the logs generated by the scheduling process to facilitate troubleshooting.The Disaster Tolerance module is used to avoid situations where the dispatcher is unavailable due to node downtime or process crashes.After actual deployment,the dispatcher has been running online on a cloud platform in China,providing services for container scheduling of large data,AI,simulation calculation,gene sequencing and other high-performance jobs.Compared with the original scheduling method of Kubernetes,the coarse-grained scheduling method solves the problem of job processes waiting for each other,and the effective utilization of resources is increased by 10%.Compared with the multi-tenant scheduling method divided by a single weight model,the multi-tenant scheduling method which combines dynamic resource allocation and preemption improves cluster resource utilization by 30% and inter-tenant fairness by 40%.Compared with the random scheduling method,the GPU topology-aware scheduling method improves the average job execution efficiency by 2.5times. |
