File-type-aware In-Network Redundancy Deduplication Schemes

With the development of network technology, there appear a lot of file-sharing Internet applications. Redundant and similar data exist in the mass data that are uploaded by users. Uploading all files directly to servers results in the low efficiency of network bandwidth and high-latency of response time. Therefore, the key of sharing Internet applications is to select and upload the unique files in the uploaded data. In this paper, we propose a file-type-aware in-network deduplication scheme, which exploits temporal and spatial localities to quickly detect duplicate data.The idea behind file-type-aware in-network deduplication scheme is to implement file-type-aware in-network deduplication in the software-defined network(SDN). We use software-defined network to replace the traditional network, to ensure the portability, flexibility and extensibility. Deleting the redundant data in the first-hop network device can fully exploit temporal and spatial localities of data. Files are preprocessed in the client to prepare for in-network deduplication. We implement the in-network deduplication by extending Open Flow switches' matching rules and processes. We use a memory cache space and add a module to detect and delete duplicate data on switches. We also achieve global in-network deduplication by extending the function of the controller. We keep the fingerprints of files instead of the original files to improve the caching efficiency of switches and the matching speed of redundant data.Theoretical analysis and experimental results demonstrate our method can effectively and efficiently reduce the network bandwidth and improve response time. Meanwhile, experiments show that our scheme has portability, flexibility and scalability.