Research Of Migrating Loading On Reading Of Hadoop Namenode

With the improvement of information technology, the data volume has become bigger and bigger, people need a better storage system.Local file system has been unable to meet the people's demand. In order to meet people's demand of performance, capacity and scalability, distributed file system emerge as the times require.HDFS is a typical distributed file system. Through the analysis of HDFS, it is concluded that under reading and writing in a large number of small file, Namenode will face a huge load, it will become the bottleneck of the system. As we all know,standby Namenode nearly has the name structure with active Namenode,so we can reduce the server load through standby Namenode. Dynamic hash algorithm is used to distribute metadata.The same hash value of the file gathered in the same entry as a migration unit. The dynamic balance algorithm is used to determine the migration of the entries with the historical load and the current load, which ensures the relative balance of the load of the active and standby node. We can use the client to update the hash table entry to ensure the synchronization of the hash table.Test results show that after choosing the reading load migration design, the Namenode performance have improved significantly, with the increase of the number of visits to the system, our system have obvious advantages.For more than 1000 concurrent read access, the Namenode performance increased by 15%. When lots of reading access to small files, our design improve the performance of 1 times. Relative to the static subtree partitioning and static hash segmentation, the load balancing is better. The system shows good expansibility when backup node extantion.