An Image Compression Framework With Assisted Inpainting

As a fundamental technology to multimedia computation and communication,image compression,especially lossy compression that targets natural images,has been continuously studied in recent years.Existing image compression schemes can be categorized into signal-processing-based and vision-based.By virtue of time-frequency as well as other signal analysis methods,mainstream signal-processing-based schemes regard the statistical redundancy among pixels as the adversary of compression.Through decades of development,such schemes have been quite mature and it is now difficult to further improve their performance.On the contrary,vision-based schemes target the visual redundancy inherent in natural images,and try to identify and utilize the important visual features that are extracted from images.However,their performance has been greatly influenced by the availability and effectiveness of appropriate image analysis methods.We propose a novel image compression framework that integrates newly developed inpainting techniques,which has combined the advantages of signal-processing and vision-based compression schemes and thus greatly improved the compression performance.Our proposed image compression framework works as follows.At the encoder side, partial image regions are intentionally and automatically removed;meanwhile,distinctive visual information in the removed regions is extracted and represented as assistant information,which is coded and transmitted together with an incomplete image.At the decoder side,after decoding the incomplete image,image inpainting is performed under the help of transmitted assistant information so as to fill-in the removed regions and finally reconstruct a complete image.In this framework,there are some important techniques including the inpainting with assistant information,the extraction and coding of assistant information,and the region removal.Such issues are discussed in detail so as to realize the compression framework.●Starting from the available inpainting techniques,we first investigate the form of assistant information.On the one hand,according to the partial differential equation(PDE) models,we argue the importance of image edges in inpainting; moreover,edges are easy to extract and can be efficiently coded.On the other, taken into account the similarities between image patches,we propose to utilize patch displacement information into inpainting,which leads to a non-parametric scheme.●Then,we devise two new inpainting techniques that take advantage of edge information and patch displacement information,respectively.Edge-based inpainting can be realized either by virtue of solving Laplace equations or by structure-texture propagation followed by texture filling-in.Patch-displacement-based in-painting extends and enhances the availability of non-parametric texture synthesis techniques.Compared with state-of-the-art inpainting techniques that are not aware of the assistant information,our new techniques achieve much better performance in restoring image regions accurately.●Meanwhile,we investigate the extraction and coding of the assistant information. A new edge thinning algorithm is proposed for the edge information,together with rate-distortion-optimized edge estimation for intra prediction.Patch displacement information is also extracted according to rate-distortion optimization in the non-parametric scheme.●We propose two solutions to region removal,one of which enables feedback and the other disables.In the former solution,there is an embedded decoder at the encoder side,which provides actual rate-distortion cost during reconstruction and the cost determines whether to remove a region.This is quite similar to the mode selection well-known in compression.In the latter solution,encoder directly analyzes the original image and infers the relative complexness of different image regions according to a specific inpainting method;then,encoder selects the regions with low complexness to remove them.We discuss on the pros and cons of each solution and realize both according to rate-distortion optimization.We have realized new block-based image compression systems that integrate the aforementioned edge-based and patch-displacement-based inpainting,and enable or disable feedback during region removal,respectively.Extensive comparisons have been made with standard JPEG as well as H.264 intra coding.Experimental results show that new systems with feedback can provide PSNR gain as well as improve the visual quality of reconstructed images,especially at low bit-rates.New systems without feedback achieve significant bit-rate savings at high quality.Such results demonstrate the efficiency and practical importance of our proposed image compression framework.There is some possible future work within the new framework;and the framework can be applied into video compression as well,which has achieved some preliminary results.What is more important,our initiative work reveals the huge potentials of future image compression,and we believe the smart decoder paradigm that integrates newly developed image synthesis techniques is a promising direction in the next-generation compression.