3D Morphable Face Audio-Driven Lip-Sync System

With the development of science and technology,3D face reconstruction technology is becoming more and more important in the field of computer graphics,which has a wide range of applications in human-computer interaction,movies,3D games and other fields.In the facial animation of virtual characters,how to keep the mouth movement and voice synchronized has always been a research hotspot.In order to reconstruct the corresponding mouth shape,it is necessary to carry out speech feature analysis,mouth shape feature analysis and mouth shape reconstruction.Traditional methods can not guarantee the matching accuracy and production efficiency when reconstructing the mouth shape,which makes the process of good mouth animation low efficiency and high cost.In this paper,the related technologies of voice and lip synchronization for 3D face are explored,and an automated system from voice to 3D face model generation is designed and implemented.The system consists of three modules: in the speech processing module,the speech is processed by Mel-Frequency Cepstral Coefficients technology to extract the speech features;In the processing module of the neural network,the Long Short-Term Memory network is used to map the speech features to the 2D face features;In the 3D face model module,2D face features are modeled to 3D face model using 3D morphable model technology.The main work of this paper is to improve the existing Long Short-Term Memory network,improve the accuracy of mapping relationship between voice features and mouth features,and realize the prediction of 2D face features by voice.An improved face reconstruction algorithm for 3D deformable models is presented,which improves the modeling speed from 2D face features to 3D face models.The experimental results show that the improved Long Short-Term Memory network and the 3D morphable model algorithm have higher accuracy and speed in the process of face reconstruction,and avoid the traditional high-cost and time-consuming methods.The3 D face voice and mouth synchronization system achieves the desired results with the features of low cost,high efficiency and automation.