Image Inpainting With Generative Adversary Network And Attention Mechanism

Digital image inpainting refers to the use of computer technology to restore the defective image,and use the known information in the image to infer the missing pixel value according to certain rules so that the observer cannot distinguish whether it is a defective image.Traditional image inpainting methods are usually based on texture expansion or similar patch matching.These methods all use the information redundancy in the image to be repaired to complete the repair task,and it is difficult to have a good effect in some complex inpainting situations.For example,the missing area is large,or the texture structure of the missing area is complicated.Especially for the inpainting of face images,how to ensure the overall continuity of the results while inpainting the details and texture features of the missing parts has always been a challenge.The current mainstream image inpainting methods are based on the method of generative adversarial network,through the joint learning between the generator and the discriminator,to generate more realistic images.On this basis,this thesis designs a face image inpainting method based on a generative adversarial network.The main work of this thesis is as follows:(1)Image inpainting not only requires the content of the inpainting result to be semantically reasonable but also requires the generated texture to be consistent with the surroundings.To better obtain the context information in the image to repair the missing part,a two-stage inpainting method based on the dual attention mechanism is proposed.The generator is divided into two stages.The first stage performs a rough repair,and the second stage adds a dual attention mechanism,which improves the operation mechanism of the contextual attention layer,strengthens the correlation between the generated image features,and enhances the repair model for the image.The ability to understand and predict context structure.An adversarial training model based on a multi-scale discriminator is designed so that the generator can obtain features of different scales during backpropagation training.At the same time,the spectrum normalization constraint is imposed on the discriminator to stably generate the training of the adversarial network.The model structure based on the attention mechanism and multi-scale discriminator can make the repair results look more real and the texture details are also richer.(2)Based on the above model,it is proposed to add a face parsing module for broken faces.The face parsing module can recover the missing part of the semantic information for the damaged face image,and divide it into a set of label masks that can identify the location and size of the facial features.Using the output of the face parsing model,a weighted attention mechanism is designed to restrict the range of feature matching of the attention mechanism,so that the attention mechanism only searches from the same semantic area when performing feature matching.This improvement decouples the structural reasoning and content filling tasks in the image inpainting task,which not only reduces the repair difficulty of the repair model,but also avoids the unreasonable facial features of the repair network output,or the generated location does not match the real situation.