Research And Implementation Of Metadata Management Scheme For Cloud Storage

Cloud storage based on cloud computing has become a rapidly developing solution for mass data storage and management recent years. Metadata management is critical to distributed file system. Many large-scale storage systems such as HDFS involve the decoupling of metadata management from file data access. In such architecture, a single master server manages all metadata, while a number of data servers store file data. This architecture can’t meet the exponentially increased storage demand in cloud storage, as the single master server may become a performance bottleneck. This paper presents a metadata management scheme for cloud storage. Our scheme employs multiple Namenodes, and divides metadata with improved consistent hashing algorithm based on bucket. System monitors Namenodes with heartbeat mechanism, and metadata can be dynamically migrated according to system workloads. To maintain reliability, metadata is replicated in different Namenodes with log replication technology, and Paxos algorithm is adopted to keep replication consensus. Besides, caching and prefetching technology is integrated in Client to improve system performance. Test result shows that it has a well metadata reading and writing performance with multiple Namenodes architecture and c ac hing tec hnology.