Research On FPGA Heterogeneous Cluster System Based On YARN

In recent years, research on integrating FPGA into Hadoop platform to accelerate job computation has received widespread attention. Through using Hadoop to distribute workload among multiple nodes, and using FPGA hardware acceleration within each node,the specific acceleration algorithms have obtained good effect. However, most of the current approaches are based on Hadoop 1.0 platform and assume the isomorphism between computing nodes. There are no unified management and scheduling on the cluster level. In addition, the software and hardware structure of the entire cluster is highly customized to run specific algorithm. Thus such kind of designs suffers from poor universality and low resource utilization. Moreover, limited by the specificity of FPGA's computational logic, existing methods to integrate other accelerators, such as GPU, can't be directly used to solve the problems of FPGA integration.The emergence of YARN resource management platform provides the possibility for the unified management of FPGA resources. Combining with the characteristics of FPGA accelerator, and extending the YARN platform by using the resource representation dimension expansion and the tag based resource scheduling methods, the extended YARN platform can unify the management and scheduling of FPGA resources in a heterogeneous clusters with multiple FPGA accelerators. Furthermore, the expanded YARN platform makes the location of FPGA accelerators transparent to users, and provides a unified interface for applications and users of the accelerator. Besides, by extending resource requests and sub task execution logic of computation frameworks, the extended computation frameworks have the ability to apply and use FPGA accelerators in the running process while maintaining the original calculation model unchanged.The results of running common applications as well as the FPGA accelerated ones on the extended YARN resource management platform show that the extended YARN platform has the ability for FPGA resource management and can also support the computational frameworks of native YARN platform. Therefore, the universality and resource utilization of the cluster have been greatly improved. The results of writing FPGA accelerated programs under extended framework show that the extended framework simplifies the request and usage of FPGA resources, thus enhancing the efficiency of application development. The results of algorithm speedup experiment on the cluster demonstrate that many factors has decisive influence on the speedup ratio, which include the data input speed,number of concurrent tasks on node and computational complexity of the application itself.