Research And Improvement Of Hadoop Scheduling Algorithm Based On Artificial Bee Colony Algorithm

Swarm intelligence optimization algorithm has high efficiency and convergence speed in solving NP problems, which has got the attention of more and more domestic and foreign researchers and engineering. Swarm intelligence optimization algorithm has become one of the main research directions to solve the cloud computing resource scheduling problem. At present there are two main directions in swarm intelligence optimization algorithm, the first one is to carry out an in-depth study of some intelligent algorithm, improve and optimize the algorithm by improving its own shortcomings and absorbing the features of other intelligent optimization algorithm. The second one is to produce new algorithm by combining them through different ways and learning from each other’s strong points. In this paper, we start from these two directions to optimize the resource scheduling of cloud computing platform, selecting the Hadoop cloud computing platform to improve the resource scheduling efficiency of Hadoop cluster.The purpose of this paper is to schedule the resource of Hadoop to multiple tasks and find the minimum of task completion time by improving the intelligent optimization algorithm or combining with multiple optimization algorithm to generate a new algorithm. At first, compare several mature optimization algorithms and analyse the advantages and disadvantages of each intelligent optimization algorithm,it is found that compared to and other algorithms, artificial bee colony algorithm is less sensitive on the dimension of the problem and suitable for solving high dimension problems. It has advantages of less control parameters, strong robustness, fast convergence speed and has obvious advantages to solve the problem of cloud resource scheduling. Therefore, in this paper, the following 2 algorithms based on the artificial bee colony algorithm:(1) Research on Hadoop job scheduling based on improved artificial bee colony algorithmIn view of the problem that the artificial bee colony algorithm is easy to get struck at local optima, first introduce Gaussian mutation idea, to enhance the local search ability of artificial bee colony, and then introduce the adaptive factor, adjust the bee colony optimization strategy dynamically to speed up the search speed, improve the search ability, optimize the cloud computing resource scheduling strategy, improve the resource utilization rate and reduce the task completion time.(2) Research on Hadoop job scheduling based on differential colony hybrid scheduling algorithmFusing two or more algorithms of swarm intelligence optimization algorithms according to a certain strategy will play their own characteristics, the actual effect is more than any single optimization algorithm. Therefore, this paper puts forward the differential-colony hybrid scheduling algorithm which is used to schedule the resource for large-scale parallel computing. The two algorithms can play their respective advantages, reduce the convergence time of the algorithm and the number of iterations, achieve a stable and effective optimal solution, and maximize the efficiency of cloud computing resource scheduling.Finally, the Hadoop cluster is built and used to do some contrast experiment to verify the improved artificial bee colony algorithm and differential colony hybrid scheduling algorithm. Through the analysis of many experiments, the most suitable parameter values are selected to ensure that the algorithm achieves the best performance. Through the experiments, the performance of the optimized algorithms are tested and compared with the original scheduling algorithm of Hadoop. The results of the analysis and comparison of the experiments are concluded that the improved algorithm and the hybrid algorithm can shorten the completion time of the job and improve the efficiency of the cluster to a certain extent.