Research And Application Of Job Scheduling System In MapReduce Based On Hadoop

Hadoop was developed in recent years to deal with large data platform specifically, it is an open source distributed data processing framework, with interms of dealing with large data reliability, scalability, efficiency, scalability, low cost and other advantage. As Hadoop MapReduce job scheduling system, it uses the idea of divide-and-conquer, together with groundbreaking design ideas to realize its functions so that it can handle data larger. Although MapReduce has a unique advantage in job scheduling, but still in the process of scheduling there are performance bottlenecks, for example, long works hours in the implementation, jobs takes too long to affect the user experience, low system resource utilization, and other defects. Thus making it more equitable and efficient algorithm of job scheduling has become an important research and industry.In order to solve some of the Hadoop job scheduling system performance issues, and based on the due data into account, presented issues, and based on the due data into account, presented the Effective Sequence(ES), the Most Effective Sequence(MES) generator, MES updater and exception handling concepts and techniques, was designed based on the most then select MES as job scheduling policy proposed MES updater to the algorithms, so as to avoid increasing the burden on system overhead. Abnormal clusters considered, MES’s exception handling mechanisms can be found in the cluster nodes and abnormal operation, so that you can quickly make a corresponding solution. Internet data as test data sets used, system test for ASAMES. Results show that ASAMES the average completion time for the job, jobs, with an average completion rate system throughout and performance than the original system and job scheduling algorithms have greatly improved.By applying ASAMES to job scheduling system, can effectively reduce the average execution time of the job, and improve system throughput, thereby enhancing the Hadoop platform job scheduling process performance.