Research And Improvement Of Load Balancing Optimization Under The Hadoop Platform

In the cloud computing and big data environment,load balancing has become one of the focuses of the research.Load balancing is one of the main goals to achieve the optimal scheduling in the cluster.The unbalanced load of computing nodes can lead to the decrease of computing efficiency and serious waste of resources on the cloud platform.When the task scale is relatively large and the load of many nodes is relatively high,further optimizing the scheduling algorithm can effectively avoid the load imbalance between cluster nodes.In this paper,the load balancing mechanism of Hadoop cluster is studied,then the partition algorithm and the intelligent algorithm are improved respectively,in order to enhance the efficiency and performance of the cluster.The main contents include:(1)Research on dynamic load balancing algorithm based on improved partitioning strategyAccording to the partitioning algorithm of Hadoop platform does not consider the data non-uniformity caused by the data density.This paper proposes to increase the partition granularity and add dynamic feedback mechanism in operation,in order to make full use of idle nodes to balance high load nodes,which can ensure load balancing,at the same time,can improve resource utilization.(2)Research on load balancing optimization based on dual swarm fusion intelligent algorithmBy using the advantages of the two intelligence optimization algorithms,we can overcome their shortcomings and improve the efficiency of the existing algorithms.Therefore,this paper make use of the excellent global search ability of ant colony algorithm and excellent lateral search ability of bee colony algorithm to combine the two intelligent algorithms to propose a dual swarm fusion optimization algorithm.So that they can fully play their respective advantages,balance the load of the cluster,improve the utilization of the cluster resources,enhance convergence efficiency,and shorten the task execution time.Finally,by building the Hadoop cluster environment,the above two algorithms are implemented on this cluster and the performance of the improved algorithm is compared with the original algorithm.The two improved algorithms can effectively balance the cluster load,improve the resource utilization of the cluster and shorten the completion time.