Optimization Design Of MapReduce Based On SDN

With the development of Internet, more and more applications have led to a large number of data processing requirements. To concurrently process big data,MapReduce with an implementation named Hadoop is proposed. However, in practical Hadoop systems one kind of issue that vitally impacts the overall performance is known as the NP_complete minimue makes span problem. The main reason for this problem is the long time in the process of calculation data migration since bandwidth is scarce resource. Software Defined Networking(SDN/OpenFlow) is the most representative structure, which is a kind of new innovative network architecture. SDN technology completely strips out the control plane from switch and migrate it into the controller, so that OpenFlow switch is only responsible for data forwarding. The SDN network structure is programmable and the controller can know the global network topology, which let us see the possibility of the MapReduce network optimization in the OpenFlow network structure.Thus, we first analyzes the main reason which affects the work performance of MapReduce, that is the data from the Map node to the Reduce node transmission takes very long time. In order to solve this problem, we try to make the Map intermediate data combined on the switch, and the bandwidth can be saved by merging data. Base on the above ideas, we put forward the “OF-MR” system. It composed by the OpenFlow network structure and Hadoop node. In this paper,we add the MapReduce routing calculation based on SDN component, and extend the operation of coding and decoding data packets on OpenFlow switch.Finally, we set up the OF-MR system and do a lot function test and performance evaluation to demonstrate the feasibility and effectiveness of the design.