Researches On Map-Reduce Scheduling Algrithm In Heterogenerous Environments

Cloud computing technology enable small and medium sized enterprises don't have to set up their own data centers. By using"Pay-per-use", they can also fulfill the needs of computing and storage. What behind the cloud computing system, are the powerful parallel computing and distributed storage technologies. Map-Reduce is the parallel programming model and scheduling approach of current cloud computing system. In heterogeneous environments, traditional Map-Reduce scheduler is inefficiency. It causes system resources'wasting, with long response time, and low throughput.In the beginning, this paper briefly introduces the definition, development, application scenarios, key technologies of cloud computing as well as an improved Map-Reduce scheduler: LATE. The main work of the paper is the proposing of an adaptive Map-Reduce scheduling approach. According to drawbacks of traditional Map-Reduce scheduler and LATE scheduler, the adaptive Map-Reduce scheduling approach provides methods of improving. In the proposed methods, the scheduler dynamically adjusts the time proportions of each stage of Map and Reduce tasks according to historical information. And, the scheduler further classes all the nodes, for launching backup tasks of slow tasks on fast nodes. Thereby, reducing response time and trying best not to leave nodes idle, so as not to waste system resources. This paper also analyzes why the proposed methods are helpful, and the deficiencies of other scheduling methods. Through specific experiments, this paper verifies that the method of learning from historical information can decrease the response time by 15% in the test cluster. And by using all the methods which has proposed in this paper, the response time will decrease about 25%. The creative points of this paper are as follows: (1) this paper introduces historical information into Map-Reduce scheduler for the first time. The scheduler dynamically adjusts the time proportions of each stage in the lifecycle of Map and Reduce tasks according to historical information. (2) This paper considers Map task's two stage character for the first time. (3) This paper further divides slow nodes which proposed in the LATE scheduler into 2 types: Map slow node and Reduce slow node. (4) Each node collects historical information by itself.