Research Of Efficient Metadata Management In Encrypted Deduplication

With the rapid development and progress of information technology and the global surge of information from users,governments,and companies,they are all facing enormous storage pressure and the increasing overhead for storage.Chunk-based deduplication technology has emerged.With the help of this technique,the storage system now only needs to store the small-size references to that duplicate chunk,which can effectively suppress storage overhead by 98%,especially for the backup storage system.However,the chunk-based deduplication system requires higher security due to the increasing need for outsourcing and special usage,which needs to add another encryption layer to the current deduplication system.Therefore,to encrypt data chunks,the message-locked encryption algorithm has been widely used in order to apply deduplication and suppress storage effectively.Still,the deduplication system needs the fingerprint of encrypted data chunks.However,the existing work can not tackle the several issues,including how to suppress metadata storage overhead from both file recipes and key recipes,and avoid high computational overhead and management on keys.This thesis proposes two schemes to address the aforementioned issues.The details of the schemes are as follows:1.Efficient metadata management-Metadedup.Metadedup builds on indirection to suppress metadata storage overhead by assembling the metadata from data chunks to form metadata chunks,followed by applying deduplication to that.In the meanwhile,file recipes and key recipes now store the metadata from metadata chunks.Due to the high redundancy and similarity of backup datasets,it is provenly feasible to apply deduplication to metadata chunks.The experiments show that Metadedup has metadata storage savings up to 97% for FSL and VM datasets,and 67% for MS datasets.It incurs no more than 2% for index overhead.In addition,the experiment also shows that it can only save 0.22% by combining Metadedup with compression algorithms,which can prove that the Metadedup is efficient enough.As for the performance,this solution achieves an averaged write overhead 6.88% higher than CDStore.2.Secure and distributed key management-Metadedup X.This scheme builds on top of Metadedup,so it is also efficient in reducing metadata storage overhead.In the meanwhile,this scheme combines the similarity-based technique with server-aided MLE to solve the offline brute-force attack against the MLE key of data chunks.This scheme also implements the OPRF protocol on key generation for higher security and privacy.In order to mitigate the high computational overhead,this scheme distributes key generation to all servers.The experiments show that the prototype can still achieve 73.33 MB/s for writing and 109.06 MB/s for recovery.In terms of storage efficiency,even with the similarity-based technique,it can still save metadata storage by at least 88.94%.Metadedup X can achieve a certain degree of load balance while memory skew rate is less than 0.91% and disk skew rate is less than1.05%.