Face Synthesis Using 3D Model And Generative Adversarial Network

Face synthesis is a long-standing challenge in the field of computer vision,before the popularity of deep learning,the classic face generation method is to map the 3D face model to the 2D space,and edit the attributes by controlling the model.With the rapid development of deep learning and computer vision,as well as the increasing demand for real-world face generation,A technology named Generative Adversarial Networks which can generate fake faces has been proposed,and on this basis,the realism of the synthetic face is continuously improved.With the gradual rise of concepts such as virtual reality and the metaverse,people hope to drive a false face by themselves,so that they can protect their privacy while conducting virtual social interaction,which pose a huge challenge for face synthesis.In face synthesis,the method of driving the source face to change its pose through the target face’s actions and expressions is called face drive.Aiming at the problems of poor consistency of movement,difficult to maintain identity features,and distortion when pose changes in face synthesis in current driving algorithms,this thesis proposes a face synthesis scheme based on 3D models and generative adversarial networks under driving tasks.Innovations in framework and model implementation.The main work is as follows:(1)A set of end-to-end face driving system has been built,and innovations are made in algorithm framework and model implementation by introducing 3D models and generative adversarial networks.The system consists of feature extraction module,motion module,rendering module and identification module.In the face synthesis stage,first input the target face to the feature extraction module to extract action information and map it to the linear latent vector space,and then input the latent vector and source face to the motion module to generate the optical flow field.Finally,input the source face and optical flow into the rendering module to obtain the driving face that retains the identity characteristics of the source face and the pose of the target face.Experiments show that the driving face synthesized by this algorithm is superior to the current mainstream algorithms in terms of feature similarity,perception similarity,expression similarity,pose similarity and other indicators.(2)From the perspective of face feature disentanglement to improve the driving algorithm.This thesis proposes to optimize the sampling method of 3D face parameters and the insertion method of hidden vector after mapping and separate the pose information for feature replacement to guide the synthesis of faces when the pose changes.This reduces the entanglement of identity and action features in the latent vector linear space,and improves the model’s ability to learn face features and robustness to pose changes.After the improvement,this method has improved in the indicators of feature similarity and pose similarity,and has achieved the purpose of optimization.(3)A fully automated face-driven solution is implemented.The driving of the source face can be directly realized under the condition of inputting a single source face image and the target face driving video.Under the condition of inputting a single source face image and target face driving video,the drive of the source face can be directly realized,and the source face can be edited by modifying the three-dimensional parameters to edit the expression,action,etc.of the source face without the target face.This thesis also provides a large-scale speaker dataset after expansion,cleaning,and enhancement,and the corresponding face feature parameters reconstructed from the 3D model.