Research And Implementation Of High Concurrent Opportunistic Resource Allocation In Hadoop YARN

Efficient resource management to improve the throughput in large-scale cluster has become a research focus with the rapid development of applications of Big Data.YARN,as the new generation of resource management system in the version of Hadoop 2.0,is more efficient in resource utilization,task processing performance in cluster and capable of handling more kinds of workload than previous systems in Hadoop.Due to the fact that a task usually occupies more resources than it actually uses during some stage of its life cycle,a relevant amount of resource is idle and can not be allocated to satisfy the requirements of pending tasks.Therefore,the high concurrent elastic resource allocation is studied in this dissertation based on the.exiting resource management of YARN.In order to address the deficiencies of resource allocation in YARN,this paper presents a high concurrent opportunistic resource allocation strategy named Ballon,which can dynamically adjust the configured resource of a node depending on the actual resource utilization in the operating system of the node.Moreover,Ballon classifies different resource requests of all kinds of applications into four different types.Consequently the elastic resources can be allocated to proper resource request,so as to improve the utilization rate of cluster resources and avoid the resource conflict caused by resource elasticity scheduling.The performance of YARN cluster and Ballon cluster are assessed through a series of realistic experiments under different volume of workload.And the two schedulers were compared by using the parameters such as the utilization rate of the cluster CPU,the usage of the memory,the concurrent degree of the cluster tasks and the running time of the cluster.In the single sample task experiment,the experimental results show that our prediction approach of ResourceRequest Classifier reaches an prediction accuracy of 0.87,which is sufficient to most size-based scheduling algorithms.In addition,the scheduler shows a better improved effect for CPU-intensive applications.In the multiple sample task experiment,the experimental results show that Ballon scheduler greatly improves the efficiency of the combinational applications of CPU-intensive and IO-intensive,and it and can increase the resource utilization of cluster obviously.In sum,Ballon scheduler can effectivelyimprove the concurrency of tasks in Hadoop YARN platform,and mitigate the severe resource contentions due to resource over-provisioning.