Working Principle And Applied Research Of MapReduce

In the past50years，along with the computer technology integrating into the sociallife, the information explosion has accumulated to a extent of initiating a reform. Itnot only makes the world full of information more than ever, and its growth rate isaccelerated. Subject of information explosion, such as astronomy and genetics,created the concept of “Big Data”. Now this concept is applied to all areas of thedevelopment of human intellectual. In the modern society, the amount of informationhas a rapid growth rate and these information accumulate a great amount of data. It isexpected that in2020,more than1/3of digital information produced per year willreside in the cloud platform or deal by cloud platform. We need to analyze andprocess these data, in order to get more valuable information. MapReduce is aprogramming model to deal with “Big Data” in the present.First of all, this paper introduced the background and the research sisuation ofHadoop. The Paper described the project structure of Hadoop as well. Then, itanalysed and described the architecture of HDFS, and introduced its access strategyand security mode. After, focused on the analysis of the MapReduce programmingmodel, in view of some kind of MapReduce algorithm, had a deep research on theworking mechanism of MapReduce. In the above process, this paper mainlyintroduced the various operations in relational algebra, such as selection, projection,intersection, difference, and, natural join, grouping and aggregation, and designed theMap function and Reduce function for the process of these operation. In addition,this paper also introduces the application of MapReduce in Hadoop, including themain execution flow of its task and the error handling mechanism for MapReduce inHadoop. Finally, the paper using three PC built a Hadoop cluster, and realized theexample of word frequency statistics and file search by MapReduce programmingmodel.