Design And Implementation Of A File Backup System Based On Source De-duplication

Along with the rapid development of the information and network technology, wecan get various information data from the network,not only improve our work efficiency,but also can enrich our life. But needs of information data storage and backup is rapidlyrising, result in bringing information data quantity explosive growth, data storage systemstorage requirements more and more larger, the network bandwidth more and more higher,the consumption of energy resources more and more serious, the data storage andmanagement costs continue to rise, so the existing data storage systems are facingmounting pressure and challenges.Although data have greatly increased, but the study found that there are a number ofredundant data in the existing data storage system, which about60%of the totalinformation. Therefore, in order to ease the great pressure that the growth of informationdata quantity lead to, reduce redundant data, make full use of the present resources andreduce the data storage management cost,the research of de-duplication has become themost popular research subject.According to the existing de-duplication data storage system whose de-duplicationprocessing is on the server, which have bottleneck problem,such as scalability is bad, thelow efficiency and is restricted to the network bandwidth and so on.This paper proposes abased on source de-duplication files backup system, which will make de-duplicationprocessing module transferred to data backup client. The de-duplication module used toimplement the file chunking and data de-redundancy, which can improves the systemscalability and the efficiency of storage and backup and greatly reduce the systemrequirements of the network bandwidth and realize the separation of redundant datadeleted process and data storage backup process.In addition, the system have client fingerprint table and index management function.The client will collect data chunk fingerprints in the process of de-duplication. Thosefingerprints will be stored to the client and the server. The client maintenance a smallsubset fingerprint storehouse which is a part of the the whole data chunk fingerprint. Theserver maintain overall data chunk fingerprint storehouse. This way can optimize the process of data block fingerprint inquires, improve performance of chunk fingerprintretrieval and further reduce the network bandwidth. Index manager will remove datachunk that is not used and release storage space.