Research And Optimization Of Reliability Of Hadoop Distributed File System

As rapidly growing of new produced data, the security and computing performance of data have been given more and more consider. So, many new technologies occurred in recent years. And Cloud Storage and Cloud Computing are the newest research direction.More and more companies and research institutions have chosen to use Hadoop to build their Cloud Storage and Cloud Computing, including Yahoo!, Facebook, IBM, etc. Hadoop mainly rely on the Hadoop Distributed File System to treat and store data. That's why so many companies see Hadoop Distributed File System as the research foundation of Cloud Storage and Cloud Computing.This paper detailed the treating process of HDFS and its replication mechanism. HDFS provides high security and availability as each data has many copy among different datanodes. Although HDFS has many datanodes, it only has one metadata server which is the bottleneck and will cause single point failure problem. This paper designed a distributed system which is based on Paxos consensus algorithm to resolve the single point failure problem. And we also designed an election mechanism in order to improve the security and performance of the system. There are two roles in the system after election, which are Leader and Follower. And there has only one server as Leader and the others are seen as Follower. The Leader works as the specified acceptor and learner to coordinate and synchronize the data among all Followers. If there have N metadata servers in the system, it can tolerate at most ( N ? 1) / 2 metadata servers of failure.As the test result shows, the designed system can work if N/2+1 metadata servers is up. Compared to the failure of Follower, the failure of Leader affects a lot more of the performance of the system as it needs to elect a new leader while the failure of Follower doesn't need.