Research On Dynamic Resource Allocation Method For Multi MapReduce Jobs In Cloud Computing Environment

With the popularity of cloud computing technology, Hadoop becomes more users' selection to process big data. In Hadoop, resource management is always one of the hot issues. Effective resource management, not only can effectively reduce the average response time of the system and improve user satisfaction, and improve the performance and interactive capabilities of the system, but also improve the utilization of resources.Based on in-depth study of Hadoop resource scheduling algorithms, his thesis proposes a dynamic resource allocation method for multi MapReduce jobs which aims at solving the problem of statically configured resource in the current Hadoop system which can't fit the personalized needs of jobs. This method is composed of initial resource allocation and dynamic re-allocation for multi MapReduce jobs.It dynamically configure the resources assigned to jobs to ensure jobs'completion time and resource utilization of the cluster. Firstly, for initial allocation,based on historical data of job execution, establish a mathematical model of job completion time prediction, and estimate resource requirements of jobs according to the jobs'completion time goals and the information of jobs initialization, and give the concept of customer satisfaction about its resource allocation, based on the principle of fairness in customer satisfaction, taking the maximization of system resource utilization into account, establish resource allocation model, and gives a heuristic algorithm for the initial allocation of resources. Secondly, for dynamic reallocation,capture the dynamic changes in the remaining execution time of the tasks periodically, improve completion time prediction model to predict changes in resources demand of every job, and perceive whether there is a new job submitted to the system, and propose the concept of event of changes in job resources demands, and based on the occurrence of an event of changes in job resources demands, according to the goal of fairness in customer satisfaction and resource utilization, dynamically reallocate resources for jobs.Based on the above research, this thesis implements a resource scheduler for Hadoop and performs three jobs that include Sort, Combine and Select, and verifies the validity of dynamic resource allocation method for multi-MapReduce jobs proposed in this thesis by the comparison with Fair scheduler, on the completion time of jobs and resource utilization of system.