Research On Metadata Management In Object-based Storage System

Object-Based Storage Systems (OBSS) adopt a new interface—object interface, which is an effective integration of fast direct access and scalability in block-based interfaces and high security and cross-platform data sharing in file-based interfaces. Object-based interfaces can provide more rich semantics than other interfaces. The basic access unit in an OBSS is an object. An object contains not only user data but also attributes that describes access characteristics of user data.In a large-scale OBSS, metadata accesses are very frequent, and MetaData Server (MDS) is the potential performance bottleneck. Thus, we need to study the issue of high-performance and scalable metadata management. In an OBSS, the data placement strategy not only determines how a file is mapped into one or multiple objects but also assigns an appropriate Object-based Storage Device (OSD) for each newly created object. Meanwhile, it is also used to locate the right OSD storing accessed objects. Additionally, it has a critical impact on the system performance, and a reasonable data placement strategy is desired in terms of the system scale. Metadata in MDS record the corresponding relationship of files (or directories) and objects, and the loss of metadata will cause data not to be able to be accessed. Therefore, the reliability of metadata is very critical.A novel distributed metadata management strategy is proposed to provide high performance and scalable metadata service through four techniques, including directory conversion metadata, mimic hierarchical directory structure, flexible partition methods targeting metadata of diverse characteristics, and the application of database to metadata backend. Firstly, on the basis of in-depth analysis of the metadata composition structure of the user component in conventional file systems, an improved method for metadata storage and management is proposed, which improves the metadata access speed due to high transaction throughput of database. Secondly, each metadata is stored as a record in the database, and directory data storing the mapping relationship of file names and inode numbers are not stored in any persistent storage (such as disk), and an indirect scheme is employed to mimic the hierarchical directory structure, which can avoid the structure itself becoming a hot-spot and thus it can provide high performance and scalable metadata service. Thirdly, directory conversion metadata are adopted to avoid the directories traversal in subtree partition shemes and file metadata migrations incurred by renaming a directory in HASH schemes. Consequently, it can improve overall metadata access performance. Lastly, based on different characteristics of each kind of metadata, different partition methods are adopted to facilitate the expansion of the system scale. The experimental results show that the strategy can significantly improve the metadata access performance and the scalability of the system.As the metadata workload changes over time, a static distribution of metadata workloads in MDS cluster may cause a MDS to become the system performance bottleneck at a certain time. Load balancing is implemented in the MDS cluster to provide high performance and scalable metadata service. A load balancing algorithm is introduced to solve the load imbalance in the MDS cluster. It adopts metadata request response time as the metric, and attempts reducing the difference of metadata request response times among all MDSs to achieve load balancing in MDS cluster.A data placement strategy is proposed for small, determinate scale storage systems, which utilizes the genetic algorithm to solve the placement problem based on different file characteristics, and it endeavors to seek the approximate optimal solution. A group-based differentiated location strategy is proposed for large, volatile scale storage systems. It firstly arranges each OSD into a storage sub-cluster according to the period of the OSD joining the system, and a distributed algorithm is adopted to map objects into different sub-clusters. Different objects are placed with different methods in a sub-cluster. A heuristic method is employed to select a lighter load OSD to place a large object, and the advanced hash algorithm is adopted to place a small object. The strategy takes into account not only the flexibility of the object distribution but also the system scalability, and the experimental results show that the strategy has an outstanding performance and scalability. The advanced hash algorithm is a new distributed algorithm, and it is proposed based on the variation trend of the OSD scale in the sub-cluster. It not only inherits the merits of a simple hash algorithm that the computation overhead is small and objects can be uniformly distributed among OSDs, but also ensures that a small amount of objects migration overhead is induced to maintain the correctness of the algorithm when the number of OSDs changes. A novel scheme is presented to enhance the metadata reliability of the system by adding an attributes page for each user object, which makes full use of the expressive object interface. Leveraging a Markov chain, the metadata reliability of the method is analyzed. The scheme requires no additional hardware equipments, and it also does not exclude other schemes of improving the metadata reliability of storage systems. Therefore, it is a good supplement for achieving higher metadata reliability.