Job Scheduling Research Based On High Performance Computing System Log

In high performance computing cluster, cluster management system plays an extremely important role. Job scheduling is the core of cluster management. Job scheduling involves the whole process of the job from submission to completion, and related to the various parts of the cluster. By designing and using of an efficient job scheduling algorithm, we can make full use of high performance computing cluster system resources to ensure that the operation is fast and executing efficient. Job scheduling is the relationship between job to resources, and the use of resources generate the system log. The performance of the scheduling strategy is directly related to the load characteristics of the job, and the performance characteristics of the job scheduling can be further improved by extracting and analyzing the system log. Simulation is an effective method for the study of complex systems. In the process of research and evaluation of the scheduling algorithm, using the job scheduling simulator to build simulation environment can play a multiplier effect.In this paper, based on the job log analysis of high performance computing system, a multi queue backfill job scheduling algorithm is proposed, and a simulation framework is built to simulate the performance of the system. The main research contents include:(1) The high performance computing cluster of BGI is studied. The characteristics of the system and job scheduling are summarized through the analysis of the actual working log of the high performance computing system. Based on this, a simulation framework is built by using Grid Sim simulator to simulate the cluster.(2) For the existing problems and defects of the job scheduling strategy, the algorithm of the backfill scheduling algorithm based on the optimization of memory resources and multi queue load balancing scheduling algorithm is proposed. The backfill scheduling algorithm based on memory resource optimization is improved by traditional backfill scheduling algorithm. Multi queue load balancing scheduling provides a queue level load balancing, and proposes the improvement scheme: the dynamic selection of the threshold strategy and the optimal migration strategy.(3) Experiment simulation is carried out using the actual system’s operating log in the simulation framework. Simulation results show that the proposed scheduling algorithm is better than the traditional job scheduling algorithm. Therefore, based on the scheduling strategy, it can effectively reduce the job waiting, improve the system’s throughput, the job response and system resource utilization has a better effect.