An integrated framework enhanced with appearance model for facial motion modeling, analysis and synthesis

Human faces provide important cues of human activities. Thus they are useful for human-human communication, human-computer interaction (HCI) and intelligent video surveillance. Computational models for face analysis and synthesis are useful for both basic research and practical applications. In this dissertation, we present a unified framework for 3D face motion modeling, analysis and synthesis. We first derive a compact geometric facial motion model from motion capture data. Then it is used for robust 3D non-rigid face tracking and face animation. One limitation of the geometric model is that it can not handle the motion details, which are important for both human perception and computer analysis. Therefore, we enhance our framework with appearance models. To adapt the appearance model to different illumination conditions and different people, we propose the following methods: (1) modeling illumination effects from one single face image; (2) reducing person-dependency using ratio-image technique; and (3) online appearance model transformation during tracking. We demonstrate the efficacy of this framework by experimental results on face recognition, expression recognition and face synthesis in varying conditions. We will also show the use of this framework in applications such as computer-aided learning and very low bit-rate face video coding.