| Hadoop is a software framework that supports parallel computing.It has unique advantages in the distributed storage,extraction,analysis and calculation of data,which makes it attract people’s attention in the field of big data processing.Since Hadoop 2.0introduced YARN as its unified resource management system,the cluster has made great breakthroughs in task allocation,resource monitoring and data sharing.YARN has three built-in resource schedulers.However,with the extension of the application,these schedulers are not satisfactory to users in terms of cluster resource utilization and system throughput.Therefore,it is important to study how to rationally allocate and schedule resources,optimize and improve the scheduling performance of YARN resource management system for massive data calculation and processing.In this thesis,the resource scheduling mechanism of Hadoop YARN has deeply analyzed.From the perspective of how to improve the overall resource utilization of the cluster and reduce the overall task execution time of the system,aiming at the problem that the reserved resources exists in YARN can not be fully utilized and the speculative execution mechanism predicts the inaccurate startup task,this thesis proposes a corresponding solution.Firstly,in order to allocating resources more reasonably,a multi-dimensional constrained genetic algorithm based hadoop resource scheduling method is proposed.This method uses the heartbeat mechanism of YARN to obtain the information of CPU speed,memory size and load of nodes to initialize chromosomes.At the same time,the crossover,mutation and replication of the algorithm has been improved,and the dual fitness function has been introduced to ensure the convergence and validity of the algorithm.Extensive experiments show that the scheduling algorithm designed in this thesis can effectively reduce the execution time of the cluster by about 20%.Secondly,for a more reasonable resource reservation,a progressive non-blocking opportunity resource reservation mechanism has proposed.In hadoop cluster,if a cluster node cannot satisfy a resource request for a task,it will reserve resources for it.YARN adopts a resource guarantee mechanism based on incremental resource reservation,which makes the overall resource utilization of the cluster low.In this thesis,a node is selected as a resource reservation node based on the priority of nodes,and then a non-blocking opportunistic resource allocation strategy is used to allocate resources to nodes.Experiments show that the progressive non-blocking opportunity resourcereservation mechanism can effectively improve the cluster resource utilization and system throughput,thus shortening the task execution time.Finally,an inferential execution mechanism based on time series is proposed for abnormal tasks.The thesis analyzed several types of abnormal tasks in Hadoop cluster,discusses the impact of abnormal tasks on the overall task completion time,and proposed an improved backup task startup mechanism.In order to prevent backup tasks from starting too much,which results in overload of the cluster and affects the overall performance of the cluster,this thesis establishes a prediction model to evaluate the necessity of backup tasks starting.What’s more,the final experiment shows that the improved scheduling algorithm can greatly improve the overall resource utilization of the cluster significantly reduce the execution time of tasks on each node. |
