Research And Implementation Of The Key Technologies Of Cloud Storage Metadata Management

With the rapid growth in the amount of global data and data-intensive applications, cloud storage systems are more widely used. To build a cloud storage system, distributed file system is essential. Current mainstream distributed file system uses the mechanism that separates metadata and data management, storing files metadata in the metadata node while data in the data nodes. Obviously, the performance of metadata node is a key to the availability of the whole storage system.Firstly, this thesis researches on the key technology in the architecture for a distributed file system in cloud storage. It discovers that many mainstream distributed file systems use a single node management system for all metadata. This architecture is easy to run into system performance bottlenecks in the face of massive data environment in cloud storage, because of the limitation of system memory in node itself, high volume of certain hot data and network bandwidth. Then this thesis analyzes the characteristics and main technical in metadata management strategy, and does an in-depth research on critical metadata management issues.There are some shortcomings of current metadata management strategy:(1) Renaming metadata and modifying directory will cause a lot of metadata migration and network overhead.(2) Due to a large number centralized access of customers at the same metadata in a high-density, a centralized single MDS becomes access hotspots. This causes load imbalance, and affects system performance. Metadata cache can significantly reduce the interaction between the metadata server and database server, thereby reducing the system response time and improving system performance. For the above problem, we propose a feasible cloud storage metadata management solution referred to the existing mainstream distributed file system. The related work of the improved solution as follows:(1) Uses the distributed nodes of a name cluster instead of a single name node.(2) Propose a distributed algorithm based on pseudo-random sequence and adirectory of granularity distribution metadata.(3) Puts the metadata cache technology into metadata management. Then presentsthe concept of metadata retention, which is calculated based on the characteristicsin cloud storage. Also, a replacing and write-back policy is proposed by using theseindices.From the result of simulation experiments, we find that the distributed algorithm based on pseudo-random sequence could significantly improve the balance of the metadata distribution, and can also ensure the optimal amount of data migration statistically. The execution efficiency of the algorithm is also very impressive.And we also find that the new cache replacement policies this thesis proposed not only can enhance the cache hit rate, but also perform with higher stability and flexibility compared with the traditional replacement algorithm. Therefore, it effectively improves the efficiency of cloud storage metadata management.Then this thesis introduces the design and implementation of cloud network disk system, and applies the above metadata management strategy study results in the implementation in order to verify its feasibility.