A Study Of Scheduling Algorithm Based On Hadoop

Although the Hadoop cluster has existed for many years, it is still the main tool to accelerate computing process and is widely used in famous Internet companies. It is of great significance to make research on Hadoop since the scheduling algorithm plays a key role in cluster and the scheduling algorithm of Hadoop still needs to be improved. Cloud computing technology allows users to quickly obtain clusters of arbitrary size to execute a given workload and to rent resources in a “pay-per-use” manner, which means it is meaningful to study how to reduce the rental cost of Hadoop cloud platform. At the same time, with the continuous expansion of the cluster scale, it becomes more important to make research on the fault-tolerant ability of node.In order to use the limited resources to maximize reduce the completion time of jobs, a new speculative execution algorithm based on balance, we called Balance-SE, is proposed in this paper after making research on speculative execution algorithm of Hadoop cluster.. When the speculative execution is running in Balance-SE, every job will be screened firstly and only the job that satisfies the condition can run on the speculative execution. As a consequence, some unnecessary backup tasks execution will be avoided and the completion time of jobs will be reduced indeed. At the same time, as the cloud platform allows users to rent resources in a “pay-per-use” manner and in order to minimize the cost when users rent a cloud platform resource, an optimal Hadoop cluster configuration for every different Map Reduce application must be determined. Here, the configuration includes the number and types of virtual machines and the job schedule. In order to minimize the rental cost, a completion time target is given for a set of Map Reduce jobs, and a homogeneous or heterogeneous Hadoop cluster configuration is determined for all those jobs. Because of the importance of Hadoop cloud platform’s fault-tolerant ability and the fact that different types of VM have different computation power, the fault in different types of VM will lead to different cost. As a result, an adaptive fault tolerance algorithm is proposed in this paper. Every VM will update its own reliability according to the situation that whether it completes its job or not. If one VM completes its job, then its reliability will be increased and otherwise its reliability will be decreased. The VM whose reliability is less than the given threshold will be set a checkpoint mechanism and different types of VM nodes will be set different checkpoints mechanism. As a result, the fault-tolerant ability of system will be improved.In order to verify the effectiveness of the algorithm designed in this paper, experiments are made according to the above two algorithms and the results shows that the Balance-SE algorithm proposed in this paper can Reduce the completion time of jobs well compared to Hadoop speculative execution mechanism and LATE algorithm.On the other hand, the adaptive and checkpointing fault tolerance algorithm based on decision platform algorithm will improve the fault-tolerant ability of system and can reduce the completion time of jobs.