Research On Job Management Middleware Based On Cluster

With the rapid development of cluster technology, the clu ster job managementsystem has become one of hot topics in the field of High Performance Computing.However, the job management system is generally configured and managed via thecommand line, requiring better knowledge and skills, which has many disadvant ages,such as complex to use, difficult to learn, easy to occur errors, etc. How to use thesehigh performance computing resources more conveniently and effectively hasbecome a common concern for both service providers and users. In this paper, wefocus on the design and implementation of the cluster job management middleware,based on one of pratical projects about Tianhe-1supercomputing platform, and try toreduce the difficulty of using high performance computing resources for thoseordinary users and improve resource utilization. The main works are as following:(1)In this paper, we firstly explore the definition, architecture and classificationof cluster techniques. we particularly focus on analyzing the architecture ofTianhe-1supercomputer’s hardware and software. And then, we aim to probe the theCluster Job Management System, especially about the architecture of the SLURMJob Scheduler, its job priority computing strategy and how it works.(2)According to operational requirements of the super-computing platform, thispaper proposes a job scheduling strategy based on customer value. This paper firstlyintroduces the background and studies the Analytic Hierarchy Process (AHP) methodwhich is used for multi-criteria decision-making, and then designs the correspondingevaluation system for customer value, and uses the AHP method to get thecorresponding weight of each indicator, and finally realizes the scheduling strategybased on the middleware and does some experimental tests. Experimental resultsshow that the new job scheduling strategy can reduces effectively the average waittime and the average turnaround time of those jobs submitted by higher-value userswhile ensuring the overall performance of the system.(3)This paper designs and implements the cluster job management middlware.This paper firstly introduces the functional requirements of the middleware systemand the key technologies used in the implementation of system, and then introducesthe design concepts and the structure of data table s related to the job management,and finally realizes the system. Meanwhile, this paper designs and implements aCAE Application Packaging approach based on Template(CAE-APT). The approach is based on XML, which is platform-independent, easy to maintain and extend. Theprototype development of the job management middleware has been completed. Weconduct some some joint tests by working with the portal subsystem and clustersubsystem and the result shows it runs well.All of experimental results show that t he research and optimized works on thethe cluster job management system in this paper have effectively improved theefficiency of job scheduling and resource utilization of clusters and can meet variousresource needs for users of different customer values. Furthermore,the middlewareworking with portal subsystem not only provides user-friendly interface, but alsoimproves user’s efficiency. All these benefits illustrate that our proposed algorithmsand systems will be widely used in practical applications.