Facial expression analysis on manifolds

Facial expression is one of the most powerful means for people to coordinate conversation and communicate emotions and other mental, social, and physiological cues. We address two problems in facial expression recognition in this thesis: global facial expression space representation and facial expression recognition method with objective measurement.; We propose the concept of the manifold of facial expression based on the observation that the images of all possible facial deformations of an individual make a smooth manifold embedded in a high dimensional image space. To combine the manifolds of different subjects that vary significantly and are usually hard to align, we transfer the facial deformations in all training videos to one standard model. Lipschitz embedding embeds the normalized deformation of the standard model in a low dimensional Generalized Manifold. Deformation data from different subjects complement each other for a better description of the true manifold. We learn a probabilistic expression model on the generalized manifold. There are six kinds of universally recognized facial expressions: happiness, sadness, fear, anger, disgust, and surprise, which we explicitly represent as basic expressions. In the embedded space, a complete expression sequence becomes a path on the expression manifold, emanating from a center that corresponds to the neutral expression. The transition between different expressions is represented as the evolution of the posterior probability of the six basic expressions.; These six kinds of basic facial expressions comprise only a small subset of all visible facial deformation. To measure the facial expression recognition rate precisely in the manifold model, we developed Regional FACS (Facial Action Coding System). FACS encodes facial deformation in terms of 44 kinds of Action Units (AU). By learning the AU combinations in 9 separate facial regions, the number of combinations of regional deformations is dramatically decreased compared to the number of combinations of AUs. The manifold of each facial regional can be considered as the sub-vector of the whole manifold. The experimental results demonstrate that our system works effectively for automatic recognition of 29 AUs that cover the most frequently appearing facial deformations. The FACS recognition results also lead to high recognition accuracy of six basic expression categories.; The main contributions of this thesis are: (1) A probabilistic model based on manifold of facial expression can represent facial expression analytically and globally; (2) The Regional FACS system provides a novel FACS recognition solution with objective measurement.