Research And Implementation Of The Scalability Cloud Storage Based On The HDFS

HDFS,the distributed file system of Hadoop, is one of current popular cloudstorage solutions. The amount of data, especially the small file data, need to beaddressed by HDFS clusters in practical application is increasing, and petabyte of datais common. However, the single point failure and single point performance bottleneckproblem because of a single NameNode architecture of the HDFS is difficult to copewith the rapid growth of the data volume. In order to solve these problems, HDFSmust have high scalability, so as to improve the system’s storage capacity and theability to process large scale concurrent access. So it is necessary to conduct in-depthstudy of the scalability cloud storage based on HDFS and propose implementationscheme.This paper presents a multiple Namenodes scheme based on NOSQL(Non-relational database).The program migrates the metadata originally stored in theNameNode memory to the MongoDB distributed database cluster, which greatlyreduce the work load of NameNode, thereby improving the operating efficiency of thewhole system. In addition, this scheme implements the high scalability of NameNodeand database, which can improve the system’s storage capacity and massiveconcurrency processing ability by increasing the number of NameNodes and thedatabase server.Finally, performance test of the proposed scheme is conducted through thesimulation experiment.The experimental results show that,compared with theexisting multiple NameNodes program based on MYSQL, the proposed scheme cangreatly reduce the response time of processing large-scale concurrent request, therebyeffectively improving the ability to process large scale concurrent access.