Research Of Job Scheduling Method For MapReduce

As a kind of high performance, mass storage and high reliability computing technology,Distributed computing has become one of the main methods to solve the problem oflarge-scale data processing. As a kind of parallel and distributed programming model,MapReduce can reduce the difficulty of parallel programming in distributed computing byhiding the underlying details, promote the development and application of distributedcomputing in large-scale data processing effectively.Although the existing algorithms algorithms have improve the MapReduceperformance of parallel computing effectively, But all the improvement of them only pointat one of the elements whitch effect the quality of task allocation in Scheduling processing.There is a problem of single standard in task allocation and Considering the the elementswhitch effects of job scheduling insufficiency, whitch reduce the effective utilization ofcomputing resources and system throughput in distributed computing system.Based on theformal analysis of job scheduling in MapRecue, this thesis analysis on these two problems,propose the MapReduce job scheduling algorithm based on degree of hungry in multijobenvironment.The algorithm using degree of hungry to describe the demand degree of job totasktracker in multijob environment, and realizes the rapid distribution of tasks by thecomparation of the task executive cost in different tasktrackers, maximize the systemthroughput and minimize the completion time of a single job.In order to verify the feasibility and effectiveness of the algorithm, this thisis simulatesthe process of scheduling in MapReduce based on a software whitch is designed anddeveloped by Microsoft Visual C++6.0. Based on these, this thesis designs two simulationexperiment include the task allocation model and job scheduling algorithm in multijobenvironment. The analysis of experimental data proved, by comparing and analyzing ofcommon job scheduling algorithms, the job scheduling based on the degree of hungry hasobvious advantages on the aspects of reducing the execution cost, shorten the operation taskcompletion time and improving the system throughput, whitch achieve the expected goal ofalgorithm design. Meanwhile confirm the feasibility of the ideas in this thesis fromexperiment point of view.