| With the upgrading and development of the Internet, Mobile Internet and Informationtechnology, companies are increasingly aware that the importance of information whichmeans data plays a decisive role for the development of enterprise. After entering the eraof Big Data, explosive growth of the data makes data de-duplication gaining more andmore attention not only from academic but also the commercial areas. De-duplication rateis one of the important factors which should be considered for the storage system withde-duplication. Extreme Binning, which is based on file similarity, may fail to identify andremove significant amounts of redundant data when the files lack similarity. Chunkfragment is another urgent problem for de-duplication system, and it can affect the readperformance of the system which makes the recovery performance poor.In order to further improve the de-duplication rate, this paper designs and implementsa new strategy, Segment Index. Different from Extreme Binning, Segment Index is basedon the similarity of segment rather than the traditional file. It can better unearth thesimilarity among data chunks, and provides better de-duplication rate while requiring lessresources for the system. Policy CFL (Chunk Fragmentation Level) is proposed toimprove the read performance of the system by calculating the fragmentation level andrewriting some of the duplicate chunks.Comprehensive tests have shown that Segment Index can remove the duplicate datawith the rate from93.02%to99.91%, while under the same conditions, Extreme Binninghas the rate from85.15%to97.46%. With CFL, the read performance has improved by therate of approximately58.7%comparing with no rewriting strategy. |
PDF Full Text Request