Research On Load Balancing Based On Yarn

In rescent years,the constantly increasing data volumes and varieties requests more on the computer's storage and computing,distributed storage and computer technology begins to rapidly develop.Due to the purchase time,hardware and software configurations,there are some differences in the performance of each computing node.However,as the most widely used distributed platform,Hadoop YARN is designed without consideration of nodes' different performance.it is inefficient for complex computing.For alleviating load imbalance,Yarn should optimize its scheduling algorithm and adopt new scheduling policy.This paper firstly introduces the process of installing Hadoop on Physical server,deeply studying on Hadoop source,sorting out the HDFS of Hadoop,MapReduce,Yarn and execution principle of each scheduling algorithm in the interior of Yarn and drawing the operation process of each part,supply a basis for the subsequent research on Yarn.Then this paper presents a new evaluation index for nodes.It synthetically considers parameters of the hardware configuration and the dynamic performance of the system to improve FairScheduler,and realize the task allocation based on the various nodes performance.The test on Hadoop cluster shows new evaluation index for nodes and the better FairScheduler are effective to solve the problem of load imbalance and increase operation's efficiency.Finally,adding dynamic feedback mechanism to the original FairSchduler's dispatching algorithm,it forms a dynamic feedback dispatching algorithm.The debriefing from heartbeat of Map phase will give feedback to resource scheduling system to adjust next tasks.The test results show that,dynamic feedback dispatching algorithm is functioned to work out load imbalance.