Design And Implementation Of Data Deduplication On The Cloud Gateway

With developing of the information age, big data storage transfer has becoming animportant problem that must be solved. Cloud storage provides a good solution for bigdata storage, however, due to the lack of cloud storage application programming interfacestandardization, which limits the application of cloud storage seriously.The lack of cloud storage application programming interfaces standardization makescloud storage gateway becoming an essential important component for the cloud datastorage, it is largely because of the cloud gateway acting as a translation betweenapplications and cloud provider APIs protocol. However, the cloud gateway can’t supportfor enterprise’s many other services, so most of the current cloud gateways are used forarchive and backup. The problem of the cloud gateway is that it increases the cloudstorage complexity and limits the performance at the same time, so for insurance ofperformance and simple point of view, it is not an ideal way for the main application.The purpose of designing and implementing a function for data deduplication oncloud gateway is for solving the high redundancy problom on backup and archive files. Byusing data deduplication on cloud gateway, network bandwidth between the cloud gatewayand cloud platform can be reduced, as well as the data capacity on the cloud platform.Data deduplication on cloud gateway are consist of several step: dividing a file into blocks,calculating every block’s fingerprint, seeking this fingerprint in the hashtable, if thefingerprint is existence, discarding this block, updating the file information and nonexisting blocks in the Swift cloud platform independently.Through performance compare between normal cloud gateway and cloud gatewaywith data deduplication function shows that: data deduplication on cloud gateway canreduce storage capacity about76%~91%in the Swift cloud platform for high redundancydata, and saving the70%~86%network bandwidth at the same time. The datadeduplication function improves the performance of cloud gateway with little effect on theresults of response costs.