Research On Task Scheduling Algorithm In Heterogeneous Cloud Environment

In the context of the rapid development of cloud computing,the combination of big data technology and cloud platforms is growing closer and closer.The resources of big multi-cloud service systems are dynamic,heterogeneous,and uncertain.These characteristics will reduce system resource utilization and affect task execution efficiency,resulting in lower-than-expected service quality of the system.Hadoop is a hot big data processing framework,and the scheduling algorithm is the core of the Hadoop framework.However,the current Hadoop resource scheduling framework and scheduling algorithm are not well applied to heterogeneous resource clusters.This thesis explores the dynamics of resources in heterogeneous cloud environments,analyzes the shortcomings of the current Hadoop YARN resource management framework and scheduling algorithms,improves scheduling strategies and algorithms,and designs resource quality grading strategies and dynamic evaluation algorithms.(1)A resource quality classification strategy JECS based on dynamic joint entropy is proposed.This strategy applies the joint entropy between the dynamically changing attributes of resources to divide the resources into different levels,and improves the resource scheduling priority in combination with existing scheduling algorithms.The FAIR scheduling algorithm and the Entropy4Cloud scheduling algorithm were improved through the JECS strategy,and simulation experiments were performed in a heterogeneous environment.The results show that the strategy can significantly improve the stability of the scheduling algorithm.(2)A dynamic evaluation scheduling algorithm EWS based on entropy weight method is proposed.The algorithm uses the entropy weight method to improve the Entropy4Cloud scheduling algorithm,and determines the resource scheduling order based on the comprehensive resource evaluation results.Comparing the performance of EWS,FAIR and Entropy4Cloud scheduling algorithms,the results show that EWS has better performance in terms of efficiency.Combining EWS algorithm and JECS strategy to construct JE-EWS two-level scheduling algorithm,comparing the performance of JE-EWS and EWS in simulation tools,the results show that JE-EWS has higher job execution efficiency and stability.(3)Improve Hadoop's YARN resource management framework,by adding resource information collection and resource status assessment modules,and optimize the FAIR resource scheduler.In Study I build a heterogeneous virtual machine cluster based on the OpenStack platform,and deploy the improved Hadoop framework of YARN in the cluster.The trail results show that the improved YARN framework improves the resource utilization of the Hadoop cluster and effectively improves the efficiency and stability of job task execution.