Research On Data Transmission Deduplication Strategies In File Synchronization Service

As globalize Internet society bring people abundant information, it also causes usmuchdifficultiesincopingwithsuchmassivedata. Inordertomanagethesehugeamountofinformation,cloudstorageiscreatedanddevelopingveryfast. Becauseofbeingwidelyused and focus of many companies, file synchronization service has been an effect toolfor users to manage their data in the information society as individual application of cloudstorage, and it also has become a hot research issue focusing by industry and academic.Based on the redundancy of specific data set, data deduplication can improve the storagecapacity utilization, reduce network transmission bandwidth consumption and economizeIT resources cost, and it has already been one of key the technologies optimizing cloudstoragesystem. Thedifferencingalgorithmisanotherkeytechnologyoptimizingnetworktransmission in cloud storage system because it can detect the redundancy among thedata between the two sides of the network and only transfer the difference, which canalso improve the efficiency of network bandwidth and reduce latency of synchronizationoperation.To achieve the goal of constructing well structured, clear user interface and efficientnetwork transmission file synchronization service, we make research on data deduplica-tionanddifferencingalgorithm. Theamountandcreativepointofourworksmainlyfocuson these aspects:1.Considering the prevail structure of the file synchronization service, we developEaSync file synchronization service, and the author is mainly responsible for designingand implementing the EaSync client.2. Weproposeadifferencingalgorithm,calledS-Rsync,whichcouldavoidrequiringchunk digest information form server before the client initialize synchronization opera-tion, and reduce bandwidth consumption, relieve server load. An adaptive differencingsynchronization scheme is also stated.3. We have analyzed and compared the current data deduplication technology andsystem,anddeterminedapplication-awaresourcededuplicationtechnologycanbeadoptedby EaSync Client.4. We propose DS-dedupe deduplication mechanism, which combine source dedu-plicaton and differencing algorithm, and can improve the client storage capacity utiliza- tion, and reduce the transmission amount data via network. we also outlet the details ofdesign and implementation strategies.5. We have implemented S-Rsync differencing algorithm and DS-dedupe dedupli-cation system, and compare them with other redundancy elimination strategies with ex-periments, such as Rsync and S-dedupe. The experiment results shows that, S-Rsyncand DS-dedupe can efficiently improve the storage capacity utilization, reduce bandwidthconsumption and the operation latency of EaSync server.