Research And Implementation On YARN Scheduler With Sensing Task

Along with the popularity of the Internet and the rapid growth of large Internet companies,a large number of Internet applications have mushroomed in people’s lives.The de-facto criterion of big data era Hadoop YARN(Yet Another Resource Negotiator)has also been successfully applied to more and more business scenarios.YARN has greatly improved the efficiency of batch task execution.But now all the schedulers for Hadoop cannot allocate resource in terms of the job deadline.In this paper,we propose an on-demand scheduler for Hadoop MapReduce job with deadline.For a given MapReduce job with deadline,the scheduler tries its best to allocate enough resources to meet the deadline of job.Meanwhile,combined with the cloud platform,we set up an adaptive resource scheduling platform called ARSF,ARSF can judge whether the resource of cluster can meet the deadline of all the jobs in cluster.If not,the scheduler should request resources from Cloud platform to make all the jobs in cluster finished before deadline.In this paper,the function and performance of ARSF are experimentally tested,which verifies that ARSF can scale the cluster horizontally when the cluster runs.The experimental results show that the ARSF platform is obviously superior to the common YARN platform in meeting the application time requirements.