Research On Grid MetaScheduling System And Design And Implementation Of CSF4 MetaScheduler

The emerging grid computing technologies are enabling the creation of virtual organizations and enterprises for sharing distributed, heterogeneous and automatic resource and the cooperation. Currently, grid computing technology has already been applied in many scientific fields, such as bioinformatics and high energy physics. The organization and allocation of these cross-domain resources to complete the grid tasks with optimized scheduling algorithms is the one of the key problems of grid resource sharing, so-called metascheduling. It is needed to design and implement an efficient, flexible and scalable grid metascheduling system.This paper focuses on the study of grid metascheduling system. The main works are presented as follows:1. Analyzed resource management and job scheduling of existent grid metascheduling systems, a comparison and an evaluation of these metascheduling systems was given in three aspects: resource management protocol, scheduling mode and extensibility. Then the paper proposed the way to design the metascheduling system by dividing the system into resource management sub-system and scheduling sub-system.2. Introduced a novel co-allocation mechanism for cross-domain resources called Virtual Job Model(VJM), VJM is designed to fulfill resource co-allocation for meta-scheduling, whose key idea is to submit virtual jobs to the candidate clusters to acquire the resources for real parallel jobs. Since VJM doesn't depend on advanced reservation, it can work with most kinds of local scheduler. An average cluster response time based cluster selection algorithm and a virtual job lifetime based deadlock detection algorithm were proposed. The paper also introduced a resource reorganization mechnism to reduce the re-alloction times when breaking the resource deadlock.3. Analyzed the grid user requirements and metascheduling behaviors, proposed a metascheduling model based on the plug-in mechanism. In the model, each scheduling policy was implemented as a plug-in, user can customize or combine the scheduling plug-ins, and even develop new plug-in to implement specific scheduling policies. Base on the model, four scheduling plug-ins were also designed and implemented, they are default plug-in that supports resource virtualization, array job plug-in, grid workflow plug-in and data-aware plug-in.4. Implemented CSF4 metascheduler, there are mainly three parts: 1. Analyzed the grid resource access protocols and proposed a flexible and scalable resource management framework, an implementation of the framework was given in CSF4 metascheduler. Therr service instances for GRAM protocol, LSF protocol and VJM protocol were also implemented. 2. Implemented CSF4 scheduling plug-in framework. 3. Implemented CSF4 user interface by three ways: 1) Traditional command-line interface, 2) CSF4 Portal, which is a JSR168 compliant portal, 3. Integrated CSF4 with Opal and provided dynamic interfaces to access the virtualized grid resource.5. Introduced the uses and experiment results of CSF4. CSF4 was deployed on NBCR for MyWorksphere projects and Opal-CSF4 system. CSF4 was also used in PRGAMA Grid Testbed and Avian Flu Grid projects. The experiment results of VJM and CSF4 scheduling plug-ins were given.At last, the contents of the paper are summarized and future works are proposed.