Research And Implementation Of Data De-duplication Technology

With the rapid development of digital information and productivity, Enterprise demands for information storage are also growing rapidly. Although the price of the storage device is declining all the time, it cannot catch up with the speed of the new data production. De-duplication is one of the hottest technologies because of its ability to examine a data set and store only unique data, De-duplication technology is expected to change this situation. De-duplication can save the storage space and the network bandwidth significantly when it is used in the backup, archiving and other centralized storage systems.At present, the speed of examining redundant data is always the bottleneck of de-duplication technology. For this reason, the research is focus on promoting the speed of examining redundant data, and applying de-duplication technology to the domain of backup in order to achieve highly efficient and stable backup system, so as to provide secure storage services for data. This system is used several technologies to improve the efficiency of backup and recovery, such as multi-level query mechanism of metadata, data cache mechanism and multi-thread technology. Multi-level query mechanism uses global-based bloom filter, double index cache and disk index to promote the speed of examining redundant data. Data cache mechanism can avoid frequent disk I/O operations. Multi-thread technology can improve the parallel processing ability of the system, so as to improve the overall performance.Operation of the system results shows that the application of data de-duplication technology can efficiently improve the performance of backup and recovery, and greatly eliminate the redundant data, save the storage space and improve the operating efficiency of the backup system.