A Dynamic Multi-tiered Storage With Deduplication Based On Variable-sized Chunks

Cloud storage clustered a large number of different types of storage devices to work together through cluster applications,grid technologies,or distributed file systems to jointly provide data storage and service access functions.In recent years,the technology receiving great attention.Due to the exponential growth of global data volume,major cloud storage service providers are under tremendous pressure.Multi-tiered storage has become a viable option for achieving efficient access and cheap storage.But a single multi-tiered storage system faces the problem of data duplication.The constantly and exponentially growing of duplication erasing the efficiency and space benefits of Multitiered storage systems.The traditional solution to integrate deduplication technology and multi-tiered storage prevents the two from fully exerting their respective advantages.In this paper,we propose a storage model DAMS(Duplication Aware Multi-tiered Storage)that is based on variable-sized chunks and integrates multi-tiered storage and deduplication efficiently.Then analyze the flaws of the Adaptive Replacement Cache(ARC)policy and proposed an efficient,flexible and compatible cache replacement policy named D-ARC(Dynamic Adaptive Replacement Cache).We also discussed the advantages and disadvantages of various detailed techniques.Based on the DAMS model,we designed a widely-available storage instance named StoreX to prove the feasibility of the DAMS model in practice.And variety of detail optimization techniques are proposed to improve system operating efficiency,reduce metadata space and the hash conflict rate.Finally,two testing tools are implemented to get a more comprehensive evaluation of StoreX under different parameters,and made comparative experiments on the D-ARC algorithm to verify its feasibility.The specific work to research the demand of efficient,high-capacity storage is as follows:(1)Analyze the problem of excessive data volume,summarizes the research status of the problem,concluding locality principle and high duplication rate as two basic characteristics of incremental data;(2)Propose Duplication Aware Multi-tiered Storage(DAMS)based on variablesized chunks.As an efficient,high-capacity storage solution,DAMS optimizes the operation process and combines the advantages of multi-tiered storage and data deduplication.(3)After analyze four defects of Adaptive Replacement Cache(ARC),we propose Dynamic Adaptive Replacement Cache(D-ARC)which has three features to improve the efficiency,hits Rate,variability and compatibility.(4)We discuss the advantages and disadvantages of detail technologies related,and implement StoreX,a widely usable instance of storage system based on the DAMS.The instance involves the implementation of data segmentation and fingerprint storage,and the data scheduling is carried out by D-ARC.The details are also optimized,for example,the data size is introduced to reduce the hash conflict rate,and the data activity are jointly stored with chunk information to make full use of hash table.