Research On YARN Heterogeneous Cluster Management Method Based On FPGA Acceleration

In the era of large data,the processing power of general-purpose processor can not meet the requirements of industrial and academic research on data mining and machine learning algorithms.Accelerating the integration of equipment and Hadoop cluster has become an important trend.The current research focuses on the use of accelerated devices to implement the MapReduce programming model,or the use of Hadoop cluster integration acceleration equipment to explore the use of accelerated equipment,parallel computing power and other advantages to optimize the MapReduce computing framework.Integrating acceleration devices on Hadoop clusters requires a more generalized,more reliable,and more efficient acceleration system from the perspective of implementation.Hadoop cluster computing tasks are divided into local tasks and non-local tasks,the task of the implementation of the basic design idea is to "follow the data to go",but taking into account the efficiency of the implementation of non-local tasks will be calculated through the network transmission to the task execution node,Using the computational resources of the task execution node.When the acceleration device is not deployed uniformly in the Hadoop cluster,the YARN heterogeneous cluster,the task execution node may not have accelerated computing resources,but the existing research work may not be able to complete the calculation of non-local tasks,resulting in acceleration The program failed to run.Based on the YARN platform integrated FPGA accelerator,the solution is divided into the acceleration type calculation node and the non-accelerated type calculation node.The program is divided into the acceleration type calculation node and the non-acceleration type calculation node,and the data block copy is stored in the acceleration type calculation node,And the cluster resource scheduler only allocates the resource container of the acceleration type calculation node,effectively solves the problem of successful execution of the non-local task of the accelerated program.The generic type of application runs according to the original execution mode.The proposed solution mainly transforms the HDFS client write data process and the resource scheduler resource scheduling algorithm,verifies its feasibility and forward compatibility through experiments,and expands the impact of the extended resource scheduler on the small-scale cluster Performance for experimental evaluation and analysis.