Spatio-temporal 3D facial surface analysis for recognition

The human visual system is able to extract an understanding of the embedded features in the rubbery surface of the face rather than in the 2D picture plane. This research seeks to help specify this mysterious "configural" component. I will study the issues of realistic facial surface analysis and modeling at a higher level of detail than previous work. The ultimate goal of this research is to provide profound scientific insights and understanding into how 3D facial surfaces contain embedded features not exhibited in 2D plane images, and how face representations could best fit human vision perception. As a result, this research will lead to building a humanized system for recognizing faces and their emotions as well as an automatic system for generating lifelike virtual expressions. In addition, this research will encourage new developments in human computer interaction, psychological and medical research, both of which increase scientific understanding and modeling of the cognitive processes of human vision.;In general, a good machine learning based application should include two important components: (a) a proper feature representation and (b) an effective and efficient classification method. In computer vision especially in the field of human facial analysis, a good feature representation is usually the most important part. The feature should be good enough to represent the whole facial characteristics; it should have strong discriminative power to be separated from different classes while keeping the variations of within-class as little as possible; the discriminative feature can be easily and in an ideal scenario can be automatically extracted from raw data without human involvement; the feature space should be as compact as possible to avoid high computational cost and large memory usage, as this is usually an important requirement for a real-time computer vision system. In my Ph.D. study, I proposed the curvature type based 3D surface representation for the face model. This curvature type based facial surface representation can be applied to different modalities (2D and 3D) and objects captured from different imaging sources. As to the classification process, firstly, I used a 3D template face which is composed of 1954 vertices. This template face covers the whole facial area and each vertex of the template face corresponds to a feature point on a human's facial surface in 3D domain. Then I used this 3D template face to adapt onto each individual 3D face, which creates the correspondence of each 3D face. Secondly, a curvature type representation is applied on the vertex of each individual 3D face. This representation is proved to better describe each individual face with enough accuracy while keeping the feature space compact. Considering the dynamics of 3D facial sequences in both spatial and temporal domain, I have also developed a set of different variations of the Hidden Markov Model (HMM) classifiers to capture the spatial and temporal dynamic characteristics of 3D faces. The advantage of this classifier is that it encodes both spatial (spatial relation of different sub regions on facial surface) and temporal (facial surface movements/changes at different time/stage during expressional changes) information simultaneously and is able to capture the dynamic process of a human's facial surface accurately. Therefore, it achieves a very promising recognition result as compared to traditional 2D still images based approaches, 2D dynamic images (videos) based approaches, and the 3D static image based approaches. Inspired by the success of dynamic texture in 2D face/facial expression recognition, I extended the curvature type representation into a dynamic curvature representation and used a vertex flow to track 3D features for 3D facial expression recognition. This approach has been proved to be very effective for 3D feature representation. The dynamic curvature based approach can also solve the problem efficiently for distinguishing neutral and non-neutral expressions as well as identifying the degrees of individual expressions.;This work is the first of this kind that addressed two of the most important computer vision applications, 3D facial recognition and 3D facial expression recognition. It has demonstrated its efficacy and efficiency to solve the biometrics problem by using the 3D dynamic facial avatar representations. In the future, we expect to make our 3D imaging and analysis technology be applied in the real public scenario in order to improve the real-world applications in security, machine intelligence, human robot interaction, and biomedical research. (Abstract shortened by UMI.)...