The Framework For Dynamic Hybrid Scheduling And Resource Management Toward Cluster With Heterogeneous Jobs

With the rapid development of distributed computing,more and more applications are deployed on large-scale server clusters.A server cluster is generally shared by multiple users and runs a large number of multi-source heterogeneous jobs.These jobs differ a lot from each other on resource demand,runtime and priority.Thus,in the process of job management,good job classification method and dynamic resource adjustment strategy are significant to the promotion of cluster's task throughput and resource utilization.However,some universal cluster management systems still have some problems.First of all,the computing resource quota applied by a job is often too large,and these systems lack the detection of resource over-commitment phenomenon.Second,these systems mainly focus on management for jobs with the same type,and lack the function of analyzing characteristics of heterogeneous jobs from multiple aspects.Last but not least,these systems did not implement hybrid resource adjustment and task scheduling with the combination of job characteristic,it's hard for all jobs to gain relatively reasonable running environment.In view of the above three problems,in this paper,the framework for dynamic hybrid scheduling and resource management toward cluster with heterogeneous jobs is proposed,aiming at combining job classification result with the scheduling method based on classification result.The framework collects job running data constantly and adjusts job resource quota according to the resource over-commitment prediction algorithm.Then,it generates job characteristic classification and container matching results by classification models and matching strategies.Containers with different type will be assigned to different schedulers,and be dispatched to server node according to relevant scheduling strategies to implement dynamic hybrid and low-latency scheduling.The main contents of this paper are listed below:(1)A resource over-application detection method is introduced.A resource over-application prediction method based on TAN Bayes network is introduced aiming at reducing the phenomenon of resource over-commitment.It generates resource over-application detection model according to resource application,computing data volume,and running information of historical jobs,and adjusts resource quota for future jobs.(2)A job characteristic classification method oriented to container type matching is proposed.A job characteristic classification method oriented to container type matching is proposed according to heterogeneity of multisource jobs,which analyzes the characteristics of jobs running in heterogeneous cluster from three aspects.It classifies jobs by decision tree as well as cumulative distribution method to match jobs with different characteristics to different container types,and adjust container resources according to the result of resource over-application detection.The container type matching results are needed in scheduling process.These steps compose job preprocessing.(3)A dynamic hybrid scheduling mechanism is presented with the combination of job characteristicsAiming at jobs with various types and characteristics,a dynamic hybrid scheduling mechanism is presented with the combination of job characteristics.As for task scheduling,the mechanism executes dynamic and hybrid strategies according to container type and real-time running status of the cluster,such as conflict resolution,scheduling matching,node selecting,etc.As for resource management,it proposes the resource prediction model of CPU and memory for server node,which implements detection and utilization of idle resources.(4)A prototype system based on dynamic scheduling and resource management framework is constructed.A prototype system based on dynamic scheduling and resource management framework is constructed,which is used to evaluate the effectiveness and reliability of our framework in real application scenarios.The prototype system consists of two modules: the application management module is utilized to submit jobs and related configurations;the resource management module is responsible for setting thresholds required by the strategies of task scheduling and resource management.It also monitors resource utilization and job type distribution of the cluster.