The Research Of Delay Fair Scheduling Algorithm Based On Resource Status

Recently, with the rapid development of Internet technology, large-scale data has been emerge, the traditional way of data storage and data analysis can’t satisfy the demand for mass data storage and processing any more. The Cloud Computing Platform is a very excellent solution for the problem. This paper selects Hadoop Cloud Computing platform as the research object. Job scheduling algorithm is one of the most important key technology problems. It affects the performance of platform and UE directly.Firstly, this paper introduces the background of Hadoop, and then, researches the two key points of Hadoop: HDFS and MapReduce. Then, research the work scheduler flow. Specially, this paper goes a simulation experiments under Linux system by using C language for the MapReduce technology based on thread and message communication.Then, this paper analyzes the defects of existing fair scheduling algorithm and put forward the delay fair scheduling algorithm based on the resource status(PDFS algorithm) to improve the the static waiting timeout time and non-distinguishing the racks and other racks. Besides, the algorithm determine the waiting timeout time by collecting the resource status of cluster regularly. Therefore, it has dynamic someway. Meanwhile, in order to shorten the run time of the task, the PDFS algorithm uses different scheduling policies for different types of tasks, according to the classification and the characteristics of the task.Finnaly, the paper proves that the PDFS algorithm is effectivenes through a series experiments. And it shortens the runtime of job comparing with existing algorithms.