Research On Job Scheduling Algorithm Based On Hadoop

Cloud computing has been proposed as a new computational model which is used for process of distributed dense data, and it has developed rapidly. A large cluster makes up of lots of computers by network connections in cloud computing. Users can get all of the resources in the large cluster just through one computer. Hadoop is an open source platform of cloud computing which has many advantages such as large capacity and low cost in process of distributed dense data. Job scheduling algorithm on Hadoop manages the allocation of resources and task execution. Choosing appropriate scheduling program for Hadoop has great influence on the ability to executive. So the research on the scheduling algorithm on Hadoop has vital significance.The key point of this paper is the research on improving of Hadoop scheduling algorithm. Considering the inaccurate progress score in LATE scheduling algorithm and not choosing appropriate tasktrackers for backup tasks in SAMR scheduling algorithm, this paper proposed an improved backup task scheduling (BTIS) algorithm. BTIS algorithm can compute more accurate progress score and choose a high success ratio and low load tasktracker to start backup task.The hadoop platform verified that the BTIS algorithm could schedule jobs to execute and shorten the response time. We tested our algorithm repeatedly which used much experiment data. Contrasting the BTIS algorithm、LATE algorithm and SAMR algorithm, BTIS algorithm could obtain more accurate proportions of each stage of Map task or Reduce task and find more compatible slow tasks to start backup tasks than other algorithms, simultaneously, it improves the processing efficiency of system.