Based On The Agent - Aid Map - Reduce Architecture Research And Design Load Balancing Optimization

The Map-Reduce framework provides a creative computation schema for complextask decomposition and sub-task management in cloud computing with large data set.However, the distribution of data set in the system has great impact on the effectivenessof clouding computing. In order to solve similar problems occur in applications dealingwith intensive data set, such as distributed data mining, we propose a novel model tobalance distributed data set in the system.We extend the classical Map-Reduce framework by adding an agent level andabstracting the load adjustment request into token. In the dissertation, load adjustmentconsists of load migration adjustment and load chunk replica adjustment. In the loadmigration adjustment, agent can estimate local state from resource request tokenpreviously received and choose better routing policy for subsequent tokens so that theycan arrive at the target nodes as far as possible. In the load chunk replica adjustment,agents could dynamically adjust the number of data replica based on access frequencyin a period so as to balance the heavy load in popular nodes.The main contribution of this dissertation is developing an agent-aid system toextend the classical Map-Reduce framework to improve balance efficiency of systemload. Based on the characteristics of sub-task allocation in Map-Reduce framework, onone hand, we propose load migration algorithm to adjust the heavy load on busy nodesand an efficient heuristic algorithm for token routing. On the other hand, according tothe feature of data access in the system, we put forward a load chunk replica adjustmentalgorithm to balance the number of load chunk replica on nodes with different accessextent. Furthermore, we test the algorithm efficiency with a set of experiment. Theexperimental results show that our algorithm and system could efficiently improve theefficiency of load balancing in Map-Reduce framework.