The Design And Implementation Of Workflow Scheduling Engine In HPC Environment

Along with the high-performance computing(HPC) equipment widely used, the problems which needed to be solved on HPC resources are getting more complicated. Such problems are usually composed of a number of joint jobs, solving such problems is a implementation process of a workflow job. However, the traditional job management system of the high-performance computing resources are lack of effective support to the controlling and implementing of workflow jobs, which resulting in the usage of HPC resources has been greatly restricted.To solve the above problems, the paper studies the technique of workflow in depth, focused on the automation of the scheduling and implementation of workflow jobs, the thesis brings forward a workflow scheduling engine model applied in high-performance computing environment is presented.The design pattern of the model emphasizes the software hierarchy, the entire system is divided into five layers, namely: performance layer, control layer, processing layer, interface layer and data management layer. Each layer is of functional independence and interfaces between them are simple. In the process of achieving the model, aiming at the complex relationship contained in the workflow jobs, the thesis designs the description of workflow jobs which is based on the XML language. on the base of general researching and analyzing the attributes of workflow jobs, the data structure of workflow jobs Objects is presented. Then, the thesis proposes the algorithm related to the direct priority and the indirect priority of workflow jobs, and on this basis, the system scheduling strategy is brought out which has achieved the effective scheduling of workflow jobs. By the means of packaging the bottom orders provided by the job management system, the real-time control on the implementation process of workflow jobs can be easily realized, thus, the friendly user-machinery interactive channel is provided for the users. Finally, we analyzes the scalability,maintenance and reliability of the system, and through the actual jobs running, the running result proved the high availability and accuracy of the workflow scheduling engine.