Research On Resource Scheduling For Recurring Jobs In Cloud Computing Systems

Efficient task scheduling in cluster is essential with the growth of business demand in the Internet industry.Research shows that about 40% jobs in cluster are recurring jobs and the scheduling optimization of recurring jobs will undoubtedly enhance the resources utilization and the average JCT of cluster.In this paper,we predict the resource usage of recurring jobs and schedule their tasks according to their logic and the prediction results.The main contributions are as follows:In order to predict the occupation time of CPU and the data volume transmitted on the network of a recurring job,we use linear fitting and quadratic function fitting to analyze the log data of the resource usage which has linear relationship with its input data volume.Experimental results show that this method can predict the resource usage with the error of less than 2%.For the resource usage which has nonlinear relationship with its input data,we propose an optimization strategy for prediction algorithms,which combines the log data prediction and the error prediction.This strategy weights different log data and uses the predicted error as the correction parameter of the final prediction value.Compared with the original Least Squares and HyperLogLog Counting,it can reduce their prediction error by about 70% and 50%,respectively.This paper presents TDP,a task priority allocation algorithm based on task latency.Task latency is the maximum time that a task can be delayed without affecting the JCT of the job that it belongs to.Theoretical analysis and simulation experiments show that TDP can guarantee the JCT of high-priority jobs and reduce the JCT of jobs with lower priority as well.Based on the TDP priority allocation algorithm and the critical path in the DAG diagram of a job,we design and implement a job deadline-sensitive task scheduling algorithm named JDS.Experiments prove that comparing with scheduling strategies based on data locality,Delay Scheduling and Batch Sampling,JDS can reduce the average JCT of cluster efficiently.