| Cluster computing has come to prominence as a cost-effective parallel processing tool for solving many complex computational problems. The key to making the cluster computing work well is the middleware technologies that can manage the policies, protocols, networks, and job scheduling across the interconnected set of computing resources. The research question addressed in this thesis is the on-line job scheduling problem for multi-cluster systems. We believe that, in order to fully harness the aggregate power of cluster computing systems, an efficient dynamic resource scheduling policy is required. To this end, we propose and experimentally validate several online dynamic scheduling policies that manage multiple job streams across both single and multiple cluster computing systems with the objectives of improving the mean response time and system utilization. Based on hierarchical resource management architecture, we present a dynamic parallel processor scheduling policy. We then extend this policy for dual (i.e., CPU and I/O) resource allocation to competing parallel jobs. Finally, we incorporate a fault-tolerant approach into the proposed job scheduling policy such that both the scheduler and the applications survive compute node and network failures. As part of performance analysis, we also extended several existing scheduling policies to make them suitable for cluster computing environments. We present results of the performance analysis under various system and workload parameters that demonstrate the viability of the proposed scheduling policies. |
