Geometry-Based Neural Network For Face Editing

Face editing is to edit a given face for a specified purpose and obtain high-quality images.It plays a role in digital entertainment,face recognition and security,and it is also an important field in computer vision.The face editing work in this paper includes two parts.One is multi-view face synthesis,the other is creating talking-heads,which means synthesize plausible images or video-sequences of speech expressions and mimics of a particular individual.In traditional face editing work,3D models of the face are often used.When this type of method can't deal with large pose case,or when the expression is too exaggerated,due to the texture loss.Recent work has applied generative adversarial networks in deep learning to face editing and get good results.However,these methods often use deep neural networks to perform rough mapping of the input image and the target output image,ignoring the face geometric prior information.Therefore,when there is a dramatically change between the input source image and the desired target image,such as large pose transformation or severe expression changes,the result is still unsatisfactory and the face identity consistency cannot be maintained.These problems have brought great challenges to face editing.The work of this paper innovatively applies prior knowledge of face geometry to generative adversarial networks.In the multi-view face synthesis work,we consider that all face rotations follow a similar geometric structure change.Therefore,this paper proposes a generative adversarial network model,which models the deformation in the form of convolution offsets due to pose changes during the face rotation process,and uses the offsets as an important geometric prior knowledge to guide the model to deform image features.According to different poses,the model uses the gating mechanism to adaptively fuse the primitive face features and deformation features.Our model achieves better results and retains more details,and greatly improving face recognition accuracy by modeling deformation and using geometric prior knowledge of face rotation instead of learning direct mapping of input and output.In the work of creating talking head images,we found that the results obtained by the 3D model of the face are accurate and maintain good consistency of the face identity,but cannot handle missing texture case.The generative adversarial network is good at "imagination".Even if the textures that are not in the input image can be generate.The disadvantage is that it is often not able to maintain identity consistency.This paper proposes a two-stage network,using the 3D face model rendering results as the geometric prior knowledge of the network,and introducing an attention mechanism to filter the noise and misaligned regions of the features in the network's skip connections,which greatly reduces the difficulty of network learning and can synthesize photorealistic images.