| The development of cloud computing provides a new way for large data processing, butthe cost of cloud environment a has also brought great burden to the small enterprises. Opensource framework Hadoop has become the preferred for mass data processing. Thisframework consists of two main parts: HDFS and MapReduce. HDFS is a distributed filesystem with a single central point.In order to cut down the loss of failure, there are manysolutions of HDFS high-availability. Their core idea is to use backup machine. But thesesolutions require more time, and have the possibility of missing data. The article design a newsolution of HDFS HA which based on dual-centerual server. This solution can overcome thedisadvantage of the traditional HA solutions,and also solve the existing single-point problem:(1) Introducting the concept of cloud computing, characteristics, classification andimplementation. At the same time, descripting the open source cloud computing frameworkHadoop detailed and its basic components: HDFS and MapReduce.(2) Reasearch several popular solutions of HDFS HA, These solutions can be divided intocold standby and hot standby. Combined with the status and role of Namenode in HDFS. Thesolution based on dual central server is proposed.(3) By researching Classical Paxos, designing a simple Paxos of data synchronization inthe three-machine framework.So that the system can still provide read and write accessservice effectively with the failure of node.(4) Through research and analysis the HDFS source code deeply, modifing the partstructure of HDFS, make it accord with the Sync Framework and guarantee the service forclient.(5) Through building simulation platform, testing the performance of two center-HDFS,verified the validity and feasibility of the new scheme.This article provides new ideas for HDFS high availability, the Paxos algorithm of threemachine and data synchronization framework which are based on hot standby have certaintheoretical significance and application value. |
PDF Full Text Request