Optimization Of Network And Scheduling For MapReduce In Heterogeneous Cluster

MapReduce is a popular programming paradigm in cloud computing due to it-s excellent scalability for processing large-scale data.However, MapReduce performspoorly in heterogeneous clusters. One of the reasons is that Hadoop’s built-in load bal-ancing algorithm for Map function leads to excessive network trafc-the trafc gen-erated by speculative tasks will compete bandwidth with shufing fows. We proposea new dynamic network optimizer called OFScheduler+for heterogeneous clusters torelieve the network trafc during the execution of MapReduce jobs. The optimizer fo-cuses on reducing bandwith competition, balancing the workload of network links andincreasing bandwidth utilization. OFScheduler+consists of four parts:(1) A marking mechanism that can marking diferent types of fows generated byMapReduce. It makes use of modifying the value of the ToS bits to distinguishdiferent types of fows(2) A dynamic fow scheduling algorithm that can improve the network utilization(3) A fow-rate control scheme. It can open or close MapReduce’s load balancingmechanism according to the status of the network(4) JobTracker gets the current network state by querying the OpenFlow controller,and takes the network states into consideration when assigning tasks to Task-TrackersWe instantiate a simulator and an OpenFlow testbed for evaluation. The simula-tion results demonstrate that the proposed optimizer has a signifcant efect on increas-ing bandwidth utilization and improving the performance of MapReduce by26-64% for most of jobs in a multipath heterogeneous cluster. The experiment results show thatthe proposed optimizer can be deployed into a real environment.