Reaserch Of Scheduling Model And Algorithm Based On Resource Awareness In Heterogeneous Cloud Environment

Cloud computing,which is the popular technology of big data storage and data processing,has been used widely.Cloud computing assembles large networks of virtualized services in general,the applications in it generate large amount of data,meanwhile cloud environment needs to collect,sort,process and gather these data.With the development of cluster size and the number of service consumers,resource heterogeneity and the types of user jobs are increasing in cloud environment,how to schedule jobs of different resource requirements in heterogeneous cloud environment becomes a challenge for the modern cloud computing distributed system.Because scheduling policy affects to a great extent resource utilization and the execution efficiency of jobs in cloud environment,currently the algorithms which are applied to schedule independent jobs for homogeneous environment are obviously unable to meet the trend of the development and wide application of cloud computing.Then to develop a scheduling algorithm of independent jobs or workflows which have different requirements of resource is very necessary in large-scale heterogeneous cloud environment.The Hadoop framework is currently one of the most popular distributed platforms,the job scheduling algorithm in Hadoop is the core of the allocation of cluster resources and the control of the job execution.On the basis of the research in the scheduling model of the cloud computing,this thesis further puts forward the job scheduling algorithm for heterogeneous Hadoop cluster.The main research content of the thesis is as the follows:(1)To slove the problem of scheduling independent jobs or workflows in heterogeneous cloud environment,the thesis proposes a DRAM scheduling model based on resource awareness.DRAM model considers the heterogeneity of cloud resources and the diversity of the resource requirements of jobs.Before allocating resources to jobs,DRAM first cluster the resources and jobs,then DRAM matches the groups of job and the groups of resource according to the demands for resource of different job clusters.This method can divide the heterogeneous cloud environment into several approximately homogeneous small resource groups,improving the utilization ratio of resources and effectively enhancing the execution efficiency of jobs.(2)To overcome the disadvantages of the job scheduling algorithm in heterogeneous Hadoop cluster,the thesis futher applies the above research results to the Hadoop cluster and porposes a HRAS scheduler based on resource-aware scheduling model.HRAS clusters the slots and jobs accoding to the the thoughts of DRAM model.Meanwhile HRAS modifies the storage manner of data block in HDFS distributed file system to improve data locality of the scheduler.(3)The thesis proposes a MPL multiuser scheduling algorithm based on multi-priority list to reduce the response time of small jobs and interactive jobs and to improve the interactive ability between users and cluster in Hadoop cluster.In order to improve data locality of Map task and the speed of task distribution,MPL algorithm add the thoughts of delay scheduling and batch scheduling.Moreover MPL algorithm can be applied to the HRAS scheduler for scheduling jobs of different sizes,at the same time also can be applied to the homogeneous Hadoop cluster by itself.(4)A heterogeneous cloud environment is constructed to test the performace of DRAM scheduling model by using CloudSim cloud computing simulation software.In addition HRAS scheduler and MPL scheduling algorithm is respectively tested in heterogeneous Hadoop cluster and homogeneous Hadoop cluster.Then all the experimental results are compared and analysed with the existing scheduling strategy and these experimental results are also used to verify the overall performance of the algorithms.