Based On Feedback Scheduling Algorithms For Dynamic Load Balancing In The Heterogeneous Environment Of Hadoop Design And Implementation

Cloud computing is a general term for the next Internet-based technologies. It is the integrated development of parallel computing, distributed computing and grid computing. Cloud computing technology provides users with computing resources in the form of service, including vast amounts of Internet information and efficient data analysis. Its transparency and simple programming model brings a convenient form of development and implementation to users.Google's Map-Reduce computing infrastructure divides cloud computing into "map" and "reduce" operations. Map-Reduce encapsulates parallel computing in data processing, localization file system, network load and fault tolerance.In this paper, we introduce the basic concepts, architecture and application development of cloud computing, and make detailed studies of Hadoop, which is the open source implementation of cloud computing including its distributed file system and Map-Reduce cluster computing architecture. Moreover we make detailed analysis of the job scheduling techniques, FIFO, Fair Scheduler and Capacity Scheduler in Hadoop.We examine the performance defects of existing scheduling algorithms in Hadoop heterogeneous environment, analysis of heterogeneous CPU usage and disk I/O read and write frequency of the effects of the Map-Reduce job scheduling. This paper presents a dynamic load balancing based on feedback scheduling algorithm by improving the Capacity Scheduler. In this algorithm, the hardware load cost of the running task has been calculated, in turn to effect of the existing job scheduling strategies to improve the computing nodes' operating efficiency in Hadoop heterogeneous cluster.Finally, in cultural resource sharing system environment, we configure a simple Hadoop cloud computing cluster to test the type of an application's load type, and compare the efficiency of dynamic load balancing based on feedback scheduling algorithm with FIFO, Fair Scheduler and Capacity Scheduler. The experiments confirm our scheduling policy used in the heterogeneous cluster has a better utilization.