Research And Implementation Of Hama Resource Scheduler Satisfying The Fairness And Load Balance

With the arrival of the era of Mobile Internet and Io T(Internet of Things), the amount of data is exploding. A variety of Big Data processing technology has been widely used, the most famous is the Hadoop platform. Hadoop platform can deal with many big data problems quickly and efficiently, it also provides the simple API so that developers can carry out the development of big data applications quickly and easily. However, due to the limitations of the Map Reduce model in Hadoop, it performs not very well when dealing with the iterative problems such as graph computing, machine learning and so on. Therefore, Google realizes Pregel which is a parallel computing framework based on the BSP computing model to deal with the iterative algorithms such as web page ranking, single source shortest path(SSSP), k- means clustering and so on. Hama is an open source version of the Pregel. When processing iterative algorithms, Hama’s performance is much higher than Hadoop. Resource scheduling is the core module in Hama, but it only provides a simple FCFS scheduler. FCFS scheduler can’t meet the needs of fairly sharing the Hama cluster resource between multi- users, and it exits some detects in the resource utilization and load balance.The purpose of this thesis is to design a Hama Resource Scheduler which meets the fairness and load balance. First of all, we will research Hadoop’s two important components—HDFS and Map Reduce, BSP computring model and Pregel framework. Then we will make a deep research on the Hama parallel computing framework, mainly including: system and software architecture, BSP computing engine, the lifecycle of Hama job and FCFS resource scheduler. Finally, we will research the most widely used resource schedulers in Hadoop — FIFO Scheduler, Fair Scheduler and Capacity Scheduler.Based on the above research, this thesis presents the design idea of our new Hama resource scheduler which meets fairness and load balance, mainly focuses on the management of Hama cluster resource, the fair strategy of the resource allocation, the fair policy of job scheduling and the load balance stragegy of task assignment. Finally, through the experiment we make the function and performace testing of this resource scheduler. The result of experiments prove that the new resource scheduler can meet the needs of fairly sharing Hama cluster resource between multi- users, the jobs of different users can be scheduled fairly, it also improve the efficiency of job execution by providing a load balance strategy.