Research And Implementation Of Expansibility Oriented Cluster Architecture

With the development of information industry,especially the rapid development of the Internet,the amount of data has an explosive growth.It has been increasingly concerned by a large number of corporations that what system is used to process the massive data.In Practice,the Hadoop cluster is a very useful tool to process massive data,and has been widely used in industry.The Hadoop system adopts central control,and a single master manages whole cluster.Therefore,the master becomes the key node.Once the master fails,the entire system will stop work.At the same time,when the size of cluster reaches a certain degree,the process capacity of master will be exhausted.After that,it is impossible to expand the cluster size,the cluster's processing ability is limited.In order to overcome the shortcomings of Hadoop cluster,this paper puts forward MapReduce system with multiple control points,in which these control points will share the task load of management dynamically.With multiple control points,the cluster's stability and expansibility can be improved.This paper realized the MapReduce system with multiple control points based on Hadoop-0.20.2,and bulid an experimental environment to verify the performance.The result showed the MapReduce system with multiple control points can reduce the task running time,as well as the consumption of CPU resources on the control points.At the same time these control points can share share the task load of management dynamically.