Research And Implementation On Mechanism To Support The Parallel Computing And Dynamic Resource Allocation Of Long Time Running Task

Along with the popularity of the Internet and the rapid growth of large Internet companies, a large number of Internet applications have mushroomed in people's lives. The de-facto criterion of big data era Hadoop YARN (Yet Another Resource Negotiator) has also been successfully applied to more and more business scenarios. YARN has greatly improved the efficiency of batch task execution. However, the traditional YARN has a fixed resource after allo-cation. This resource allocation mode has some drawbacks to support the long time running task on YARN. The long time running task runs as the form of back-end services in the cluster, and the demand for resources of each parallel subtask varies during the lifetime of the application. In this case, running tasks that allocate a fixed amount of resources will not meet the resource require-ments of the application at different stages. Meanwhile, the size of Hadoop cluster is fixed during the running period, and it is difficult for Hadoop cluster to expand according to the demand of resources. This is also a lack of flexibility for resource change requirements.In order to solve the problem of the YARN resource allocation, this thesis studies the optimization scheme of resource scheduling on the data cluster and the resource scheduling mechanism of YARN, and proposes a parallel comput-ing dynamic resource allocation mechanism EDCP. EDCP combines the ad-vantages of cloud computing environment and YARN. EDCP can expand the scale of the cluster in the run-time and optimize the long time running task ac-cording to the load of each sub-task during the running time. More container resources can be created for the application and more computing nodes can be deployed for the cluster when the cluster resources are not enough. EDCP provides a more flexible resource allocation mechanism to support long time running task. This thesis does experiment to test the function and performance of EDCP. The results turn out that EDCP can expand the scale of cluster dur-ing run time. Meanwhile, compared with traditional YARN, EDCP can make WordCount, TeraSort and other computing-intensive tasks reduce the running time to 0.7 to 0.9 times.