Facial Perception: Learning-Based Face Tracking And Synthesis

Facial perception can be composed of face analysis and synthesis. As a human, he is very familiar and sensitive to human faces. A human can easily recognize different human faces and different facial expressions, he can also imagine the shapes of different human faces, and artists can draw many kinds of human faces. However these abilities which are in human's blood are very difficult for computers. Hopefully, machine learning offers a possible solution.This dissertation is mainly about one unique object - a human face. Using a learning-based method as the masterstroke, we study analysis algorithms: video-based facial motion capture, and synthesis algorithms: image-based 3D facial modeling, face image super-resolution, and facial expressional hallucination.First we discuss image-based personalized 3D facial modeling. This dissertation proposes an Analysis-by-Synthesis based personalized 3D facial modeling algorithm. Through measuring similarity between the texture synthesized and the original orthogonal images' texture, and then using this mis-registrations error to guide a local and adaptive subdivision step, we can refine the 3D facial mesh model to better preserve fine facial features, and make the output 3D facial model more realistic.Second, in order to extract motion data of each facial part from a video stream, this dissertation studies multiple facial feature tracking algorithms. We propose a Bayesian network enhanced prediction based multiple facial feature tracking algorithm and a spatio-temporal graphical model based one. The latter one combines the particle filter with belief propagation. In the first step, several independent CONDENSATION-style particle filters are utilized to track each facial feature in temporal domain. Particle filters are very effective for visual tracking problems; however multiple independent trackers ignore the spatial constraints and the natural relationships among facial features. In the second step, we use Bayesian inference - belief propagation to infer each facial feature's contour in spatial domain, in which we learn beforehand the relationships among contours of facial features with the help of a large facial expression database.Later on, in some existed video streams, the face area is too small to be tracked effectively; therefore this dissertation studies super-resolution algorithms for face images. We propose a new learning-based super-resolution algorithm for face images. In the first step, steerable pyramid is used to capture low-level local features in face images, and then these features are combined with pyramid-like parent structure and locally best matching to predict the best prior. In the second step, the prior is integrated into Bayesian maximum a posteriori (MAP) framework. Finally, steepest descent method is used to obtain the optimal high-resolution face image.Finally, the real photographs are always the most photorealistic. Based on this, in order to synthesize new unseen facial expression, we propose a new algorithm - facial expressional hallucination. Given a person's neutral face, we can predict his/her unseen expression by machine learning techniques. Different from the prior expression cloning or image analogy approaches, we try to hallucinate the person's plausible facial expression with the help of a large facial expression database. In the first step, nonlinear manifold learning is used to obtain a smooth estimation for unseen facial expression, which is better than the reconstruction resultof PCA. In the second step, Markov network is adopted to learn the low-level local facial feature's relationship between the residual neutral and the expressional face image's patches in the training set, and then belief propagation is employed to infer the expressional residual face image for that person. By integrating the two approaches, we obtain the final result.In each chapter's experimental results, we present implementations for the above facial perception - face tracking and synthesis algorithms. Some related algorithms are also progressively integrated...