Research On Face Image Inpainting And Editing Based On Generative Adversarial Networks

Face image synthesis is one of the most significant research directions in the field of computer vision,which includes many problems of computer vision such as face at-tribute editing,face inpainting,face translation,face demosaicing and face swapping,etc.Recently,face image synthesis has witnessed substantial progress due to the increas-ing power of deep learning,especially the emergence of generative adversarial networks.However,current face synthesis methods still suffer from many challenges and problems,for example,disentangled representation learning of face attributes is still a very chal-lenging task;face inpainting is also faced with the problems of realism,controllability and diversity.To solve these problems,we propose several novel approaches for face editing and face inpainting based on generative adversarial networks.This main work in this paper is summarized as follows:(1)As structure(the underlying sketch)and texture(the pixel pattern mapped onto se-mantic structure)are two indispensable parts of images,inpainting algorithms must deal with them appropriately to produce realistic results.However,many curren-t methods use the end-to-end framework to repair an image,which does not pay special attention to texture and structure.Therefore,they often generate distorted structures and inconsistent textures.Moreover,they also do not allow a user to specify structure and texture information arbitrarily in the missing region.To this end,we propose a novel face image inpainting method comprising a sketch com-pletion network(sGAN)followed by a texture completion network(tGAN).sGAN focuses on repairing the sketch structures in the missing region of an image,and tGAN generates consistent texture information in the missing region based on the sketch output by sGAN and the surrounding incomplete image.By modeling the two parts separately in a deep neural network,not only the proposed method can successfully synthesize semantically valid and visually plausible contents in the missing region,but also it allows a user to manipulate the structure attributes freely in that region.(2)Aiming at the problem that the process of face image inpainting is uncontrollable and the generated results are not diverse,we propose an attribute-guided facial im-age inpainting algorithm.This approach uses the attribute labels to control the image inpainting process,and an attribute classification loss is introduced to ensure that the generated results have the specified target attributes,which enables our proposed method to generate controllable and diverse results.In addition,the use of attribute labels also provides additional information about face attributes,which can help the inpainting model to synthesize more reasonable semantic content and improve the visual realism of the inpainting results.The experimental results show that our proposed method can not only successfully control the face inpainting pro?cess and achieve a variety of inpainting results by attribute labels,but also generate more reasonable semantic content for the missing regions.(3)Aiming at the problem that the existing face editing methods fail to accurately mod-el disentangled representation of multiple face attributes,resulting in poor editing results,we propose a novel face editing method by modeling disentangled repre-sentation of multiple face attributes with GAN(MulGAN).MulGAN simultane-ously imposes attribute label constraints on both the latent feature space and the generated images to force the model to learn disentangled encodings of multiple attributes.In order to impose attribute label constraints on the latent feature space,MulGAN first predefines a latent feature block for each attribute in the latent fea-ture space to store the disentangled face attribute information.Then the attribute label constraints are directly imposed on the predefined attribute-related blocks by the attribute label filtering way,which makes the predefined attribute-related blocks become a new,stronger representation capability,learnable labels.In order to im-pose attribute label constraints on the generated editing results,MulGAN uses an attribute classifier to restrict the generated images to own correct target attributes.By the double constraints,MulGAN can accurately extract the attribute informa-tion to the corresponding attribute-related blocks,the attribute entanglements and ambiguities can be significantly minimized in the learned representation.Finally,the aimed attributes can be transferred from a reference image to a target by simply exchanging certain regions of attribute encodings,so as to achieve controllable and diverse face attribute editing results.(4)Aiming at the problem that the existing face editing methods mainly operate on a predefined set of face attributes,lacking the capability of manipulating the attribute-independent face semantics,we propose a novel exemplar-based generative facial editing approach(EBGAN).EBGAN uses a rough semantic mask as suitable guid-ance to transfer the interested face semantics from the exemplar image to the target image in the form of region inpainting.EBGAN has two main components:percep-tual network and facial editing network.Perceptual network is used to extract the feature of the predefined editing region.Facial editing network is used to achieve exemplar-based facial editing by selectively transferring information encoded by the perceptual network.In the process of information transfer,EBGAN uses a combination of reconstruction loss and perceptual loss to ensure that the informa-tion transferred by the perception network is fully utilized.Meanwhile,EBGAN also introduces adversarial loss and the attribute label constraint to filter out the redundant information caused by rough semantic masks.The adversarial loss can filter out some redundant texture information,and attribute label constraint can fil-ter out some redundant semantic structure information.The experimental results show that EBGAN can successfully transfer any semantics of interest from exem-plar to the target image,and achieves controllable and diverse editing results.