The Research And Improvement Of Scheduling Algorithm Based On Hadoop Platform

With the rapid development of the Internet, cloud computing has become the third wave of the IT, the wave bring enterprises a same problem that mining data from TB or even PB level data, and the speed of mining data determines the development of enterprises, so enterprises want to mines data with a more efficient system. as a classic cloud computing platform Hadoop can meet the requirements of the enterprises.In Hadoop the good scheduling can enhance the cluster throughput. The job that can be divided into many tasks, each task can run on any node of the Hadoop, and the process is decided by scheduler, so if scheduling algorithm designed no well would lead to network congestion and tasks maybe unequally distributed.This paper makes a deep research of the scheduler’s principle on Hadoop platform, and the artificial bee colony algorithm is improved with the differential evolution algorithm, then analyzes the shortcomings of the three classic algorithms. A new scheduler is put forward with DABC algorithms, to assign different tasks by the real-time state of each node in the system, which can make the the utilization of the system reaches the maximum, and to build the platform to test DABC scheduler. The results show that DABC Scheduler can solve the shortcomings of the existing algorithms in a heterogeneous environment better, and improve the efficiency of the system.