The Design And Implementation Of A Model To Support Cross-Domain Synchronized Resource Allocation

Grid, which is important national network infrastructure of scientific research, educational, economic, social and national defense, is a new field of research. With the rapid development of Internet technology, Grid, based on grid concept, uses the Internet to joint the variety of geographic distribution, heterogeneous high-performance computing resources, storage resources, data resources and other special linking resources, dynamic resource optimization resources to achieve complete portfolio of high-performance. At present, the grid has been widely applied in bioinformatics, high-performance physics, geology and other areas of scientific research.Before Grid technology is widely used, parallel applications run in a single cluster environment to obtain high computing speeds; with the continuous development of grid technology, a growing number of parallel applications can run in the grid environment in order to obtain higher performance. However, past parallel applications is designed for a single cluster environment, which makes its migration to autonomy, heterogeneous, distributed grid environment, encountered a synchronous wait, deadlock, load balancing issues, especially when cross-border resource co-allocation. Over the problems, we design a virtual job model (VJM), which can manage a parallel grid operations and heterogeneous resources. This model, in a layer of meta-scheduler in Grid, can co-allocate heterogeneous resources in cross-border to avoid the deadlock. VJM doesn't depend on any resource reservation mechanism, so it can use standard GRAM protocol to collaborate with local scheduler, such OpenPBS, SGE, which doesn't support resource reservation. Meanwhile, backfilling small job can ease loading function a waste of resources situation.Virtual Job Model (VJM) is a schedule model, designed to solve cross-border allocation of resources synchronization for MPICH-G2. The VJM does not immediately identify the resources in the submitting period that to be used by a job. Contrarily, according to the job requirements of resources, VJM deliver a certain number of virtual jobs (VJ). Those VJs are used to seize resources and then the resource situation will be back to the virtual job centre (VJC). VJC co-allocate resources for a parallel job depended on resources'information. In the charge of VJC, it not only can avoid deadlock, but also reduce the waste of resources by backfilling small jobs. Taking into account the distribution grid and heterogeneous characteristics, VJM does not depend on all those who were not supported by the local scheduler functions, such as resource reservation, meanwhile, the use of VJM grid nodes do not need to install new service or changing the environment configuration.VJM, as meta-scheduling module for CSF4, provides a mechanism for the allocation of resources simultaneously, and VJM can organically combine with CSF4 through the plug-in mechanism in CSF4.In VJM, VJC is composed by virtual job pool (VPool), certificate management (CAManager), as well as virtual job allocator (VAllocator). VPool are responsible for the VJ storage containers, and the management of the life cycle, and the front of a Facade pattern, to provide resources for the user interface and distribution; VAllocator has decided to distribute a parallel job of the fleet and available VJ; VJC also includes a number of auxiliary components: VJobDeliver, VJobAcceptor.VJ was distributed by the local scheduling access to resources, at first launching the personal-gatekeeper to receive parallel job from VJC, and then to establish connectivity, personal-gatekeeper port and resource information such as the IP address registered to VJC. In order to facilitate the management of VJC, VJC and VJ has been established between a control link, which is virtual job proxy (VJP) and VJ communication links, it is through control VPool also connect using the VJP management VJ.In the realization of VJM, the job submitting command mpirun was replaced by csf-mpi-run in CSF4. After the parallel job was created by JobService and submitted to QueueService, parallel job entered the QueueService scheduling cycle. Scheduling cycle in order period, as plug-in parallel operations do not care about the order of jobs, it keeps the order of the original cohort. Scheduling cycle in the match period, the plug-in parallel jobs request resources through VJM. First, the parallel job plug-in selects suitable clusters from CSF candidate cluster for parallel job through VJM, namely, the call VPool galloc () function. In the VJM, according to the candidate clusters and parallel jobs requirement, VAllocator chooses a suitable set of situation clusters. After VJP stand for VJ in VPool was added in VPool, it said that VJ has occupied a resource. Parallel job plug-in can call valloc() function to request resources in VJM. The valloc() function in VPool uses strategy in VAllocator to select available resource for the parallel jobs. VJ is distributed to the local scheduler by VJobDeliver, and the local scheduler will be operating it as a normal job. After VJ gets the access to resources, VJ began to register. In the area of security, we configured a user certificates for VJM. Therefore, VPool can unified manage VJs, and distribute VJs depended on VAllocator's strategy. In the last Scheduling cycle, parallel job plug-in checks parallel's state, and put back the resources into VPool if the parallel has done.VJM distributes VJs to occupy resources in local scheduler, and unified manages VJs in VPool. As the top resource manager, VJM uses deadlock protective mechanism to avoid deadlock and backfills small jobs when job is idle. Thanks to VPool's management, a long-time idle VJ will be killed. That breaks the causation of deadlock between multiple VJMs.This paper analyses the existing problems when a parallel job running a grid environment, and designs virtual job model (VJM) that will support a cross-border allocation of resources synchronous. VJM is in scheduling layer, it manages grid parallel jobs and heterogeneous resources. Under the use of virtual jobs, VJM is able to run synchronous parallel jobs in the cross-border, heterogeneous resources, and avoid the deadlock, and alleviate the resource waste by backfilling small jobs. Moreover, VJM can work with almost all kinds of local schedulers via standard Grid Resource Allocation and Management (GRAM) protocol as it does not depend on resource reservation. We have implemented VJM in meta-scheduler CSF4 and validate the rationality of VJM by MPICH-g2, a parallel application.