A Fast And Secure Data Deduplication Method In Cloud Storage System

With the rapid development of internet technology,the data volume of companies and individual is also increasing accordingly at unprecedented rate.Research recently shows that by the end 2020,the global amount of data will reach 35 ZB.Big Data era coming soon,traditional data storage methods have been proved unable to satisfy the massive data storage requirements.But based on the distributed file system,the cloud computing and cloud storage technology with its high reliability,good scalability,high fault-tolerance and low cost characters which can provide reliable,efficient,low-cost computing and storage service is getting more and more attention from the users and companies.However,not only in traditional back,archive system but also in modern cloud storage system,there are more and more complex and redundant data according to many surveys.Besides,the duplicate data occupy a considerable amount,resulting the huge waste of storage space.In order to increase the space resource utilization,many organizations start kinds of researches on this topic,and data deduplication is one of the most important and popular techniques to solve this problem.The process of deduplication includes four major steps,chunking,hash-computing,hashindexing,deduplication.The main contents of this paper are as follows:(1)Because the traditional hash functions such as MD5,SHA1 are becoming more and more vulnerable,so in this paper,we put forward a new hash function SHA3 which have strong ability of collision-resistant and is especially never been employed in deduplication area.(2)Besides,the I/O bottleneck which takes the longest time during the entire process is another research hotspot,and we exploit the file similarity theory to establish a two-tier indexing model to accelerate the index procedure in this paper.(3)In the end,the traditional single inline-deduplication founded on file similarity can't attain satisfactory result,so we integrate the SHA3,content defined chunking and two-tier indexing technology together to present a fast and secure client-server-based two-layer deduplication methodTLDM and give experiment to prove its high efficiency.