Research On Fine-grained Resource Allocation Method For MapReduce Jobs In Cloud Computing Environment

With the continuous development of Internet technology and the unprecedented expansion of massive data,cloud computing has been developed rapidly.In cloud computing,the MapReduce distributed computing framework has become the popular computing model which is used to process big data.At present,in order to guarantee the execution performance of MapReduce,a lot of research work has been carried out for the problem of MapReduce resource assignment.Different methods of data placement,job scheduling sequence and resource allocation strategy can make job have different execution performance.This thesis analyses current research in the field of job scheduling,to slove the disadvantages of resource scheduling for MapReduce jobs,such as the problem of low system throughput resource utilization then a new method of Fine-grained resource allocation for MapReduce jobs is proposed.This method includes fine-grained initial resource allocation and dynamic resource reallocation.Jobtracker is the master node when the mapreduce program is running which controls the job execution order and the allocation of computing resources.Tasktracker is under the control of jobtracker,carry out the tasks which is divided by jobtracker.Firstly,in the framework,to slove the problem of static configuration of the Hadoop slot,which may result in the low resource utility rate of the the Hadoop cluster especially for the case under which the required number of reduce slots and may slots is unbalanced,this thesis replaces Map Slot and Reduce Slot with Job Slot.When allocating resources,the appropriate computing resources will be assigned to the corresponding type of the task.Then,the utilization of the Hadoop cluster can be improved.Secondly,currently the Hadoop lacks of the ability of insure the global performance of the multiple MapReduce jobs which may result in the low throughput.To solve this problem,the job completion degree is defined to be used for improving the throughput of the cluster.based on this,a greedy algorithm is proposed for maximizing the throughput and the utilization of the cluster.Thirdly,In the execution of the MapReduce job,the execution environment may be changed as a result of which the low job performance and resource utilization may be occurred.For solving this problem,the event of XX is defined based on which the usage of the resource and the new submission of the job can be detected.Then,an algorithm for finding the new allocation plan is proposed to insure the performance of the overall jobs.Dynamic apperceive resource usage in the system and the situation of new job submission,dynamic decide the number and position of resource which allocation to the jobs.On the basis of the above research,build a distributed Hadoop cluster experiment environment.Through the contrast experiment and analy the experiment results,then prove this fine-grained resource allocation method is feasibility and validity.