Research On Face Image Editing With Generative Adversarial Network

Among the research topics of Computer Vision(CV),tasks related with face images have been always attracting widespread interests.In this paper,we mainly focus on Face Image Editing,where the input is an arbitrary given face image and the output is another image in accordance with certain desired requirements by making some modifications on the original input.In specific applications,the editing requirements can be turning the gender from male to female or the age from young to old,adding diverse makeups,and synthesizing different expressions.Therefore,face image editing covers a wide range of applications and can benefit various related industries including art,photography,video editing,fashion design,e-commerce and entertainment.As Deep Learning and Generative Adversarial Network(GAN)become more popular in recent years,researches on face image editing have obtained extensive progresses by generating face images that are more realistic.Many editing tasks that used to be handled by professional persons via professional tools,nowadays can also be achieved by deep neural networks quickly and effectively,which make it possible for many new applications.However,there are some problems present in existing literatures,such as the artifacts observed in the editing results and the unrelated contents that are wrongly modified.As a result,the editing quality still remains to be improved.The most fundamental issues of face image editing are how to represent the editing requirements in a quantitative and precise way,and how to measure the difference between the editing results with the desired ones.Moreover,the results of face image editing should meet the following three properties.(1)Realism.The generated images should be as realistic as possible and indistinguishable from real ones without any obvious artifacts.(2)Correctness.The generated images should be well in accordance with the desired requirements.(3)Invariability.The generated images should not change the original personal identity,and other contents unrelated with the editing requirements should also be well preserved.For different applications of face image editing,we need to carefully design the models accordingly,leverage different methods to represent the editing requirements,apply proper and effective approaches to accurately measure the difference between the generated outputs with the desired ones.In order to achieve comprehensive functionalities of face image editing,we mainly consider the following four types of editing operations,attribute editing,makeup transfer,expression synthesis and spatial editing.Based on the properties of realism,correctness and invariability,we explore and delve into them one by one.The first part mainly focuses on general facial attribute editing.The term general means that the model can achieve editing for multiple facial attributes jointly and different attributes are represented in a unified manner.We propose two models,Residual Attribute GAN(RAG)and Divided Facial Attribute Editing(DFAE),as well as four effective mechanisms,the residual attribute vector,the identity preservation loss,the enhanced generator and the sparse attention.Compared against existing literatures,our models can achieve better performances on tasks including single-attribute editing,multi-attribute editing and attribute interpolation.Moreover,we publish a dataset that contains 27,772 anime avatars,which can be utilized to facilitate researches on facial attribute editing.The second part mainly focuses on facial makeup transfer.For female face images,makeup is a facial attribute with rich implications and involves the styles of multiple regions including face,eyebrows,eyes and lips.We propose two models,Attentional Makeup Transfer(AMT)and Disentangled Makeup Transfer(DMT).The former one can achieve high-quality makeup transfer and well preserve the unrelated contents.The latter one introduces a more general architecture to disentangle arbitrary face images into two components independent of each other,the identity code and the makeup code.Compared against existing literatures,our models can not only produce results of higher quality,but also solve different scenarios of makeup transfer,including pair-wise,interpolated,hybrid and multi-modal.The third part mainly focuses on facial expression synthesis.Expression is an important facial attribute that can be utilized to express diverse emotions.We propose Disentangled Expression Synthesis(DES)based on facial action units.We leverage disentangled representation to decompose arbitrary face images into two components independent of each other,the identity code and the expression code,so that we can freely and accurately modify the expression without altering the identity.Moreover,we propose the inverse proportion attention loss to solve the problem that the generated attention mask tends to be all-zero when applying the attention mechanism in encoderdecoder architecture.Compared against existing literatures,our model can achieve better performances on synthesis tasks for single action unit,multiple action units as well as desired emotions.The last part mainly focuses on facial spatial editing.In order to accurately control the shapes and the sizes of different facial semantic components,we propose the Semantic Editing GAN(SEGAN)based on the facial parsing mask.We decompose each face image into several slices according to the parsing mask,thus preserving the corresponding content information as well as the spatial information for each semantic components.To guarantee that the person identity stays unchanged after editing,we employ the local as well as the global reconstruction losses.Compared against existing literatures,our model can produce results of higher quality on different spatial editing tasks.