Research Of GPU Cluster System Based On YARN

The performance of GPU, a kind of highly parallelized streaming processor, hassurpassed CPU significantly in a number of applications. And coupled with thematurity of GPU programmable technique such as CUDA, general-purposecomputing on GPU becomes an important hotspot. However, Similar to CPU, it isnow difficult for a single GPU to fulfill the requirements of compute-intensiveapplications and what·s more, developing a cluster system for every GPU applicationsis too costly and wasted. Consequently, this thesis focuses on studying the techniqueof CUDA on Hadoop2and aims to build a multi-gpu system based on YARN.First of all, by analyzing YARN·s architecture, it is proven that YARN is able tomanage multiple resources effectively, and a solution of adding GPU into its resourcescheduling system has been proposed. Furthermore, combined with GPU-relatedfeatures, this thesis optimizes YARN·s resource scheduling procedure against itsdeficiency, making YARN a resource management system with the ability ofscheduling CPU, memory and GPU simultaneously and efficiently.In addition, for the purpose of conveniently migrating CUDA applications toYARN system, this thesis designs a CUDA application master againstcompute-intensive applications to reduce the cost of application parallelization. And adynamic task assignment policy is proposed for improving system·s resourceutilization and minimizing jobs time-consuming when running on YARN.Finally, by analyzing results of a large number of experiments in multi-gpucluster environment, it is proven that this system has the capability of managingmultiple resources in complex and heterogeneous environments efficiently andflexibility.