Towards Geometric Supervision And Semantic Fusion Face Swapping

Face swapping is the task of generating images with the identity from a source face and the attributes from a target image,where the attributes contain pose,expression,lighting,occlusion,background,and so on.Face swapping has great usage in privacy protection,film industry,computer games,and other pan entertainment usages.In the past,face swapping relied on lots of manual operation and cost much time and labor.With the development of Convolutional Neural Networks,the methods of face swapping which mainly contains 3D-based and GAN-based have been greatly improved,but remains two challenges:(1)How to preserve the face shape of the source image.(2)How to make the results more photo-realistic.In this work,we propose a high fidelity face swapping method,which can well preserve the face shape of the source face and generate photo-realistic results.The mainly work can be concluded as follows:(1)We propose a 3D shape-aware identity extractor and use geometric supervision to optimize the face shape of the result.Previous face swapping methods only use face recognition model to keep the identity similarity,which can not get exact information of face shape.3D shape-aware identity extractor combines 3D face reconstruction with a face recognition model.On one hand,it helps get rich shape-aware identity features,on the other hand,it can generate the 3D face model of the swapped face,which can be served as geometric supervisions.(2)We introduce the semantic facial fusion module,which uses the weak supervision from facial segmentation to make the results more photo-realistic.In the feature level,it can optimize the combination of encoder and decoder features which improves the legibility of results without the loss of identity similarity.In the image level,it can learn adaptive masks and help better preserve the occlusion and background despite the change of face shape.Extensive experiments show that our method can preserve better identity,especially on the face shape,and can generate more photo-realistic results than previous state-ofthe-art methods.