| Big data is the unique data phenomenon with the rapid development of computer technology and Internet technology.The modern society is not to imagine the speed of big data,fortunately,the development of computer technology and Internet technology,in the production of large data,but also to bring people a new cloud computing technology.Cloud computing technology to bring the big data processing capabilities,making the analysis and grasp of the large data contained in the endless information,knowledge and wisdom to become possible.Distributed computing is orders of magnitude larger engineering data is divided into a plurality of small blocks,by multiple computers that computing nodes to calculate respectively after the operation result to upload.At last,the results are combined to get the calculation process of unified data.Parallel computing refers to a general computing tasks are divided into a plurality of sub blocks,with the parallel processing power of computing nodes,assigned to different processors,the processors follow the deployment mechanism for parallel execution sub tasks,and ultimately achieve enhance the computing scale or improve the calculation speed.Hadoop distributed architecture is used in the design,it mainly consists of three sub projects: MapReduce programming model and a software framework for writing fast processing of the parallel program of data on a large scale computer cluster),HDFS(build in cheap computer cluster distributed file system)and Hadoop common(and provide basis for the overall structure of support functions).Focuses on the study of MapReduce processing framework in which to establish a to deal with reliable distributed systems platform of data the distributed parallel work to speed up the processing speed,which shows its advantage when used in processing large amounts of data in the cloud computing.Finally through the test programs to analyze the process and characteristics of the MapReduce framework in data processing. |
PDF Full Text Request