Improving Of Job Scheduler Algorithm About Hadoop Based On Genetic Algorithm

Recently, with the rapid development of Internet technology, large-scale data has been emerge, the traditional way of data storage can’t satisfy the demand for mass data storage and processing any more. The Cloud Computing Platform, is a very excellent solution for the problem which comes with big data. In so many kinds of Cloud Computing Platform, this paper selects Hadoop Cloud Computing Platform as the research object. Hadoop platform is an open source, extensible, high reliable distributed computing framework. Job scheduling algorithm is one of the most important key technology problems. It affects the performance of platform and utilization rate of system resource directly.Firstly, this paper introduces the background knowledge of Hadoop, and then, researches the two key point of Hadoop:HDFS and MapReduce. Specially, research the work scheduler flow and estimate the three existing kinds of job scheduling:In these three kinds, FIFO is unsuited for multi-work and multi-user. Capacity Scheduler and Fair Scheduler need to set many parameters for performance. It is too hard for most Administrators. And, all these three kinds, in case of short work, considering time cost of changing work, the performance is not good enough.Secondly, this paper aiming at reducing time cost of running short job, meanwhile taking account of long work, improve job scheduler. From the point of job, put all task of several jobs in a big queue, and choose the sequence of run task based on Genetic Algorithm, for avoiding time cost of short jobs. And set the litmit of queue to fit long job.Finnaly, prove the improving job scheduling algorithm is more effective than existing job scheduler in case of a large short jobs through a series experiment.