Research On Real-time Face Tracking And Animation Methods Using A Monocular Camera

Generating photorealistic facial animation using computers is an important research problem in computer graphics.In recent decades,researchers have conducted a great deal of work to solve this problem and achieved great success.In industry,such as movie or game production,people usually use some special equipment and perform complex operations and computations,to accurately track facial motions and generate highly photorealistic facial animations.However,these expensive equipment and time consuming computations are not practical for applications of ordinary users.For ordinary users,using a monocular camera to track facial motions and generate facial animations is the ceasicest and the most effective method.Current technologies on face tracking and animation based on a monocular camera still has a large gap with the technologies using special equipment in the accuracy,robustness of the face tracking,and in the sense of reality,richness of details,expressiveness of the generated facial animation.This thesis focuses technologies on real-time face tracking and animation based on a monocular camera,and conducts a deep,systematic research by proposing a series of novel methods.Our work provides an effective way for ordinary users to accurately and effectively track facial motion and to generate photorealistic facial animation.Our work has involved three key components in facial animation technologies:face model representation,facial motion tracking and digital avatar creation.Details are as follows:1.Face model representation.Targeting on the problem of the lack of expressions in current face database,we propose Face Warehouse,a 3D facial expression database for visual computing applications.Face Warehouse captures the geometry and texture data of 150 users in 20 different expressions using a RGB-D camera.Based on these captured data,we build a user-specific expression blendshape model for each user,which includes the user's 47 basic expressions described in the Facial Action Coding System.Finally,we use these 150 users' expression blendshape models to construct a bilinear face model,which can represent the face shape of different users in different expressions,and can be used in many visual computing applications.2.Facial motion tracking.We propose three facial motion tracking methods based on a monocular camera.a)Firstly,we propose a method of 3D shape regression for real-time face tracking.This method trains a specific 3D shape regressor for each user,and uses this regressor to accurately track the 3D positions of facial features in the video frames for this user.b)To remove the preprocessing step for each user in work a),we propose the displaced dynamic expression(DDE)model,a novel model in representing face shapes.Based on the DDE model,we propose a fully automatically real-time facial motion tracking method based on a monocular camera.This method can be used for any user in accurately tracking the facial motions,without any preprocessing.c)Based on work b)and c),we propose a method of' real-time high-fidelity facial motion capture.This method calculates the local facial details from the local facial appearance,and then reconstructs the high-fidelity face geometry,which contains the rich facial details such as wrinkles.3.Digital avatar creation.We propose a method of image-based dynamic avatars.Based on dozens of captured images for one user,this method constructs the facial expression blendshape model and hair morphable model for the user.These captured images,together with the constructed models,are used to build the dynamic avatar for the user.Driven by a facial motion tracking system,this dynamic avatar can generate photorealistic facial animations,which contain the rich facial details and realistic hair motions.