Research Of Job Scheduling Based On Hadoop Platform

As the most well-known Cloud computing system, Hadoop platform is an open source, extensible, high reliability distributed computing framework. Its greatest feature is that it realizes transparent parallel processing to application developers, which can let application developers develop cloud computing applications without thinking the parallel process. Hadoop provides a stable and reliable interface to applications. As a new platform, there are many places need to be improved further on Hadoop platform.Job scheduling technology is one of Hadoop platform's important technologies, which directly relates to Hadoop platform's overall performance and system resources'usage. The job scheduling problem of Hadoop platform has become a hot research. Study of existing job scheduling algorithms we find that this technology is still need to be improved.Based on the analysis above, this thesis research on the follow issues:(1) Through extensive literature survey, this paper has done a systematic and comprehensive research on Hadoop platform's architecture and core component. We dived into the Hadoop platform's job scheduling technology. And then this paper gives a detailed analysis of the existing scheduling algorithms,which are FIFO scheduler, Fair scheduler and Capacity scheduler, finally we research on the PBWRR scheduling algorithm and the PWSW scheduling algorithm through experimental data.(2) On the foundation of analyzing the given algorithms, this paper creatively proposed a Genetic Algorithm Based Job Scheduler. The algorithm can schedule the job to run automatically adapt to the execution environment, and can find a shorter completion time of jobs. This paper gives a detailed idea of this algorithm, the algorithm process, the design focus. Finally, we implemented the algorithm and did some tests. The results showed that the algorithm can be effective on Hadoop platform.