| Nowadays,images occupy a large amount of storage in data centers.To cope with the explosive growth of the image storage requirement,image compression techniques are devised to shrink the size of every single image at first.Furthermore,image deduplication methods are proposed to reduce the storage cost as they could be used to eliminate redundancy among images.However,state-of-the-art image deduplication methods either can only eliminate file-level coarse-grained redundancy or cannot guarantee lossless deduplication.Similarity-based deduplication schemes are lossless and fine-grained.But they still cannot work effectively on image data because of the hidden similarity as well as the mismatch in data dimension and granularity.Therefore,a novel fine-grained deduplication scheme for images is needed to reduce storage costs further.This dissertation proposes a new lossless image deduplication framework to eliminate fine-grained redundancy among images.It first decodes images to expose similarity,then detects similar images and eliminates fine-grained redundancy among these detected images by delta compression,and finally re-compresses the remaining non-redundant data by image compression encoding.Based on this framework,this dissertation proposes a novel lossless similarity-based deduplication system for decoded JPEG image data(called im Dedup).Specifically,im Dedup consists of two critical techniques.First,it uses a novel and fast sampling method(called Feature Map-based similarity detection)to detect similar images in a two-dimensional way,which greatly reduces computation overhead and memory access.Second,it uses a novel delta encoder(called Idelta)which is compatible with image compression encoding to compress the detected similar images.Idelta incorporates image compression encoding characteristics into deduplication to guarantee the remaining delta-compressed image data to be friendly re-compressed via image encoding,which significantly improves the compression ratio.This dissertation implements a prototype of im Dedup and demonstrates its superiority by experiments on four datasets.Compared with exact image deduplication,im Dedup achieves a 19%–38% higher compression ratio by efficiently eliminating fine-grained redundancy among images.Compared with state-of-the-art similarity-based deduplication schemes running on the decoded image data,im Dedup’s similarity detector achieves a 1.8×–3.4× higher system throughput without sacrificing detection accuracy,and its delta encoder achieves a 1.3×–1.6× higher compression ratio and a 1.5×–2.6× higher compression speed. |
