Research And Application Of Hadoop Job Scheduling

Cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources. With minimal management effort or service provider interaction, the resources can be rapidly provisioned and released. It provides large scale computing capability and data storage capacity as services by organizing massive machines and clusters. In Hadoop platform, MapReduce has become the dominant programming model in cloud-based data processing. The key of the system is how to schedule job in Hadoop. According to different needs of jobs, designing the corresponding scheduling algorithm is of great practical significance for the improving the performance of the whole system and resource utilization.This paper analyzes the frame and principles of the Hadoop platform. It first introduces the Hadoop job scheduling process in detail, then gives a brief analysis of three existing scheduling algorithms on Hadoop:FIFO. Capacity Scheduler and Fair Scheduler. The real-time scheduling is referred at the end.Through the deep research of scheduling principle, we first analyze the several key parts of completion time about job scheduling. Due to job completion time is too long and resource utilization is not high in non-preemptive Deadline algorithm, then we put forward a kind of algorithm based on the Deadline preemptive scheduling. The algorithm can not only shorten the completion time of job well, but also improve the resource utilization of the system. Finally, we effectively verify the algorithm through building cluster environment and use the ganglia software to monitor cluster and collect data. The experimental results show that our algorithm can effectively solve the shortage of existing scheduling algorithms to obtain good performance test.