Research And Implementation Of Dynamic Resource Management On Hadoop

As the coming of “big data” age, more and more data needs to be processed and analysed, which presents requirement and challenge to the storage and computing technology. Hadoop gave out a good solution to these problems with limit situation. As there was no other effective tools for big data in contrast of huge requirements, Hadoop had been widely misused. To make Hadoop used in a right way and provide basic resource management platform for multiple frameworks, the Hadoop team made a complete overhaul on the ?rst generation of Hadoop(MRv1) and produced YARN. YARN lifts the resource management part taken charge by JobTracker in MRv1 into a general resource management platform and has made great progress on resource utilization.However, there still exists wastage of resources in YARN. YARN requires every application to specify the detail amount of CPU and memory size, and tasks making use of more resources than applied would be killed. As administrators do not clearly know the actual needs of each task, so they always apply for more resources than actual demand, which result in resource waste. In addition, as many computing frameworks run as applications on YARN, they need to launch long-running service processes to accept and manage their own subtasks. The resource requirement of these long-running services vary over time, but their allocated resources keep the same, which is still a wastage of resource.Given the shortcoming of YARN’s resource management, this paper represents a elastic resource management solution: Yadoop. Yadoop is based on current optimization work on MRv1 as well as YARN’s resource allocation principle. Yadoop aims to make container, which is YARN’s resource allocation unit, have resources elastically changed according to the actual usage. This solution can adjust containers’ size to a suitable position with tasks running normally and resources not wasted. Therefore,Yadoop can have more concurrent programs running and improve cluster utilization as well as running e?ciency.Compared with YARN, Yadoop performs a good e?ciency improvement with CPU-bound applications. Jobs’ s e?ciency running on Yadoop separately can be improved by 1.5 times and 1.3 times for running concurrently. Besides, Yadoop also make great improvement on cluster resource utilization.