A Lightweight Map Reduce Model Research Based On Multi Embedded Processors

With the rapid development of the cloud computing and the multi-core processorsin recent years, the application of the parallel technology becomes more and morepopular. Some parallel development technology such as MPI and OpenMP have beenused maturely in various fields. There are many frameworks based on MapReduce byGoogle, such as Phoenix, Metis and Hadoop. But Phoenix and Metis is on account ofthe shared memory architecture. It can not be used in the distributed cluster. However,the efficiency of Hadoop clusters deployed in the embedded environment is not enough.So there is no mature distributed computing framework based on the embeddedplatform. Although the MPI+OpenMP approach is a more commonly used, but it isdifficult to realize the parallel control using OpenMP. Besides, this approach alsoresulted in the overhead of the development and the maintenance.Firstly, in order to study the imperfection of the Hadoop application, this paperbuilds a cloud computing platform based on the multi embedded processors, the parallelimage processing methods is realized. With the analyze of the efficiency, it can befound that the Hadoop used in embedded environment has some blemish. On the otherhand, the MapReduce framework of Phoenix and Metis based on multi-core platformwere studied, and some performance evaluation was carried out on the Tilera36platform to discover the strengths and weaknesses of the framework.Finally, a lightweight framework for distributed computing MPI+Phoenix isdesigned based on the research described above. The main consideration is theheterogeneity between nodes, and how to take full advantages of the processor tomaximize the efficiency of the system. Then the framework of parallel imageprocessing applications are tested to evaluate the performance of various aspects of theframework as well as to test Phoenix standard test set. The result shows thatMPI+Phoenix model proposed in this paper can improve the efficiency of the dataprocessing and the parallel executing of the programs.