The Design And Implementation Of A Distributed Structured Storage Metadata Management Module

In the era of big data, many new requirements are coming with data storage flourishing. The traditional relational database cannot meet these demands any more. Based on the research and analysis of typical structured storage systems, we proposed a POI oriented distributed structured storage system named “CBase”. POI information is stored in many storage nodes in CBase. Metadata information binds POI information to the storage nodes. Metadata management is a module of CBase, which stores CBase’s metadata. The main work and features of metadata management module as follows:1. Optimized metadata management. We designed and implemented a R-tree based metadata management module. By using the operation log, we transform a lot of random disk I/O into sequential I/O, and improved the overall throughput of the system.2. Fault-tolerant metadata. In order to ensure the reliability of metadata, we designed and implemented a consistency protocol based on Paxos algorithm. We designed the Leader Lease mechanism which reduced the network traffic during synchronous process significantly.3. Fast recovery. We designed a fast recovery mechanism by using shapshot and operation log. The system can quickl y restore its metadata after break down.4. High concurrency support. By using epoll mechanism provided in Linux platform, we designed and implemented a high-performance network programming framework that can support massive concurrency.