Programming Model And Runtime Framework For High-Throughput Computing

Common techniques for dealing with large-scale problems include high-performance computing,high-throughput computing,and big data processing technologies.Due to the use of many computing resources for long periods of time and fault tolerance requirements,big data computing frameworks such as Hadoop,and MPI which is widely used in high-performance computing,are not fully suitable for developing high-throughput computing applications.In order to reduce the difficulty and improve the efficiency of developing high-throughput computing applications,this paper improves the existing DCR programming model and runtime framework to meet the requirements of multi-task and fault tolerance for high-throughput computing.With the help of the programming model and the running framework in this paper,users only need to implement the decomposition,computation and reduction processes specified in the programming model when developing high-throughput computing applications.Features such as task execution,computing resources allocation,node management,load balancing,fault tolerance are all implemented by the runtime framework.This paper first introduces the improved programming model,including: the concept of tasks;task templates and task groups;the description of task dependencies in task groups;the message model used for communication between tasks;the principle of multi-task computing resource allocation.Subsequently,this paper describes the overall structure of the runtime framework,including the overall structure of the scheduling node and compute nodes,the queue structure,the thread structure,and communication between scheduling node and compute nodes.Then,this paper introduces the design and implementation of the multi-tasking mechanism of the runtime framework,including: management of task templates,task creation,task status,task management and implementation of task manager,implementation of task executor,design and implementation of message model.At the same time,this paper also introduces the implementation of task execution,task resource allocation,load balancing and fault tolerance.Finally,this paper tests the multi-tasking,message communication and large-scale cluster support capabilities of the runtime framework on the "Sunway TaihuLight","Tianhe-2A" supercomputers and GPU clusters,which verifies that the runtime framework can meet the requirements of high-throughput computing.