Image Inpainting Of Large Corrupted Region Based On The Representation And Inference Of Stucture

Digital image inpainting is an automatic digital image processing technology that fills the missing areas in the corrupted image.The inpainted image is required to maintain the coherence and natural transition between the missing area and the known area,which satisties human visual requirements.With the development of the information technology,digital image inpainting has not only become an important research content in the fields of computer vision and digital image processing,but also has become a frontier research topic at home and abroad.It has been also widely used in many applications,such as cultural relics protection,biomedical,games and entertainment.Traditional image inpainting methods based on information diffusion or exemplar matching can achieve good results in small missing region or the missing region with repeated texture.However,when the area of missing region in the image is too large,these methods cannot accurately infer the content of the missing part.The results are always blurry or messy.The recently developed image inpainting method based on deep learning can infer the content of the missing image by understanding the context of the damaged image,which conforming to the inpainting law of human.It provides a feasible way to repair large damaged images and can generate more accurate image semantics in specific scenes such as faces and vehicles.However,there are still several huge technical challenges when directly extending such methods to inpainting scenarios of natural image,which involving composite structures and textures.(1)Content generation: Due to various object types and complex pixel distribution of images in natural scene,the existing methods generate the images by one single forward method,which often exists serious problems of shape distortion and texture aliasing,especially at the area of object boundaries.On the other hand,the intersection of homogenous texture regions of different objects in the image tends to form rich structural information,which is a good prior knowledge and is beneficial to the solution of this "one-to-many" ill posed problem.So how to infer the structural information and utlize the structural prior to assist image inpainting are important issues to solve the problem of shape distortion.(2)Texture optimization: The existing image inpainting method based on deep learning usually adopts the reconstruction loss(L1 or L2 loss)of the inpainted image and the original image in the pixel level or the feature level as the optimization function of the network model.However,the reconstruction loss has been recognized by the academic community that the generated content has serious problem of blurry.It is difficult to obtain a satisfactory result even if combined with the generative adversarial loss,which is famous for keeping the naturalness of generated image.On the other hand,the existing information in the corrupted image has intact and rich texture details.Therefore,how to learn from this important texture prior knowledge to keep the texture details of the generated content in the missing area consistent with the existing information is the problem needs to be solved in the field of image inpainting.(3)Semantic sensing: The existing methods use a deep neural network architecture consisting of multiple convolution-normalization-activation modules and combining residual network,dilated convolution and skip connection.It has been proven its effectiveness in improving the overall quality of the image.However,there are many different objects in the natural image,and the structural properties of different objects are different.The inpainting mechanism of texture details is also different.The introduction of semantic priors may improve the inpainting accuracy of object structure and texture in complex scene.However,the semantic priors of the objects in the corrupted image are partly missing and depends on the result of image inpainting.Therefore,how to coordinate the inference of the semantic prior and image content and let the two tasks to promote each other is another problem that needs to be solved in the field.In view of the above requirements and challenges,we draw on the idea that human artists first outline and restore the structure in the restoration of images.This thesis takes image structure as the main research object and gradually introduces structural prior information,texture prior information and object semantic prior information to image inpainting.We try to research on the image inpainting from the three levels of structural inference,texture transfer under structural constraints and optimization of semantics structure.The main contribution and innovation are follows:(1)Image inpainting based on the inference and perception of explicit structureAiming at the problem that existing image inpainting methods rely on a single highdimensional feature to express complex natural scenes,which may lead to inaccurate image inpainting results,this thesis proposes an image inpainting technique based on inference and perception of explicit structure.The image structure information is obtained by explicitly extracting the edge of the damaged image,and the inpainting model of structure is constructed based on the high-level semantic feature representation of the structure.Then the complete edge image is generated to provide structural prior information for the restoration of the missing region to improve the inpainting quality of the target boundary of the result.Compared with the image inpainting method based on the overall image content understanding and generation,the proposed algorithm can effectively improve the accuracy of the recovery target boundary and can improve the PSNR of the inpainted image by 1.09 dB.(2)Image inpainting based on implicit content inference and style renderingTo improve the texture details of generated region,the existing image inpainting algorithm always match the mid-layer features from an initially generated neural patch with patches in the known region and adapting it to the most similar one.But the image content would be mismodified because the neural patch mixes the structure and texture information.This thesis proposes an image inpainting method based on implicit content inference and style rendering.By separating the structure and texture features of the image into the latent space of content and style respectively,we propose the inference method of content and the rendering method of style,which is based on the different characteristics between the structure and texture.It can solve the contradiction between content inference and texture optimization fundamentally.The proposed algorithm can improve the PSNR of the inpainted image by 0.38 dB compared with the algorithm based on texture-optimized image inpainting.(3)Image inpainting based on object perception and multitasking recursive learningAiming at the problem that the existing generation-based image inpainting methods have insufficient expression ability for each semantic object in the multi-objective inpainting of complex scene,the thesis proposes image inpainting method based on object perception and multi-task recursive learning.By introducing object semantic prior information in the image,the method provides guidance for the structure inference and texture filling of missing regions.Then,based on the analysis of the interdependence between semantic object segmentation and image inpainting,a multi-task recursive learning framework for image semantic segmentation and image inpainting is constructed to improve the image inpainting quality based on semantic prior information.The algorithm proposed in this paper can be compared to the existing method by 0.57 dB improvement in PSNR.In summary,this thesis analyzes the importance of structure in the process of artificial image inpainting and starts from the representation and inference of image structure.It introduces the structural prior,texture prior and semantic prior information at three levels of the extraction and inference of explicit structure,the representation and inference of implicit structure and semantic constraints-based inference of structure.It also proposes three methods,including image inpainting method based on the inference and perception of explicit structure,image inpainting method based on implicit content inference and style rendering and image inpainting method based on object perception and multitasking recursive learning,respectively.The thesis greatly improves the image inpainting quality of complex natural scenes and lays a foundation for the application of image inpainting from a specific scene to a general scene.