| With the rapid development of the Internet,more and more users are accessing to the Internet,a variety of Internet-related services have also provided for every class of the whole society.These services affect our lives in all aspects,varying from the small events like dining and taxi,to the system for booking railway tickets and video.In the foreseeable future,the Internet will continue to develop,and the data being produced will become bigger and bigger,therefore the processing of these data and mining of the valuable information will become very important.HDFS,whether as a basic storage system for the cloud computing framework MapReduce or simply as a large mass data storage platform,plays an increasingly important role in the current society.As a widely used large-scale distributed file system,it is very important that it is able to store data efficiently under the promise of high availability and high reliability.In this paper,we study the NameNode and DataNode thoroughly,and discuss some limitations of HDFS in the current version,then put forward the related optimization solutions.The main work of this paper is as follows:(1)Introduce some relevant knowledge of cloud computing,as well as the Hadoop framework.We will mainly study on the HDFS Namenode working principle and DataNode read and write process in detail,and the MapReduce calculation process is also being brief introduced.(2)Introducing the high availability of HDFS,and analyzing the HDFS high availability solution of the earlier version,which leads to the new addition mechanism of Hadoop 2.x,HA(Availability High Availability)mechanism.We will analyze the HDFS HA in detail,and propose a new optimization solution which is more conducive to the scalability.Then we continue to analyze the metadata consistency and the active / standby switchover.In the experiment,the optimization of the metadata consistency guarantee has been verified.In the case of two-node failure,the optimization solution greatly reduces the time of switching between the active and standby Namenode nodes,and the optimization effect is obviously good.(3)Discussing and analyzing the read and write operation of HDFS.As the bottleneck of disk read and write,this paper will propose an optimization solution of a cache layer for the HDFS.In this design,we will use Memcached to implement the cache layer,the read and write operation of data will be faster because of the cache layer,in order to achieve read and write performance improvement.In the experiment section,we will take a series benchmark of read and write tests and MapReduce test,the test results show that with this optimization,the read and write operation as well as the MapReduce jobs have a certain performance improvement. |
