Research And Implemetation Of Data Redundancy Elimination Technology For Wide Area Network

With the rapid development of imformation technology, the demands of data communicationsbetween branches of the enterprises are growing fast. Especially recent years, the limited WANbandwidth can’t meet the enterprise demand for data transmission. In order to solve the problem ofinsufficient banwidth and long network delay, the article discussed to use data redundancyelimination technology to optimize the transmission data, so as to reduce the bandwidthconsumption and improve the efficiency of data transmission.The main task of this thesis is to study the Data Redundancy Elimination technology that isapplied to the WAN data optimization system. It designes and implementes the data redundancyelimination method for WAN. The study focused on the data chunking approach using an improvedCDC algorithm which combined CDC algorithm with sliding window detection technology. Itadopted the parallel filter thoughts to screen the highly redundancy data block for further chunking.Thus, it can achieve higher data elimination radio in the same granularity size. To overcome theproblem that the metadata dictionary can’t reside in the memory altogether when the processingdata increased, the paper designed a two stage storage structure of memory dictionary and diskdictionary, at the same time, adopting a multi-stage retrieval mechanism to improve the retrievaleffciency. Finally, it realize the synchronization mechanism of the data dictionary as well as theencoding and the decoding method. Experiments show that, the Data Redundancy Eliminationsystem can provide10:1or even higher compression radio while processing the potential associateddataset which helps to save60%to90%of the WAN bandwidth.Applying the data redundancy elimination technology to WAN data optimization system caneffectively reduce the bandwidth consumption effectively and improve the efficiency of the datatransmission.