Research On Dynamic Bayesian Network Based Facial Activity Recognition

Enabling the computer to understand the human emotions in a natural communi-cation way is the development direction of Human-Computer Interaction. Human face activities contain plenty of human emotion information. Facial activity analysis includes facial feature tracking, AU (facial Action Unit) recognition, facial expression recognition, etc, and is the key step for the computer to understand the human emotions. Hence, facial activity analysis have attracted significant attention in the literature. However, practical experience has shown that recovery of facial activities is currently far from mature. For example, a great number of challenges such as scale variations, appearance variations of the object, illumination changes, image uncertain, differences between different subjects, scarcity of training data, etc, need to be solved before one can develop a robust facial activities recognition system. This dissertation proposes several algorithms to address these problems, such as spontaneous facial action unit intensity measuring, modelling the relationships among different levels of facial activities, knowledge driven prior model learning method, etc. The main contents of the dissertation are as follows:We studied DBN (Dynamic Bayesian Network) based method for AU and AU in-tensity spatial-temporal relationships modelling. Most of current research in the facial activity recognition field only recognize binary AU, and the few works that recognize AU intensities also ignore the dependencies among multilevel AU intensities, as well as the temporal information. To solve this problem, we propose a unified probabilistic model based on DBN for spontaneous AU intensity recognition. The proposed method contains two steps, i.e., AU intensity observation extraction and DBN inference. Gabor feature and HOG feature followed by SVM classification is employed to extract AU intensity obser-vation. Then we employed a DBN to systematically model the dependencies among AU and multi AU intensity levels, as well as the temporal relationships. The proposed DB-N model combines with the image observation to recognize AU intensity through DBN probabilistic inference. Experimental results show that the proposed method can improve the AU intensity recognition accuracy.We proposed a DBN based model for facial feature points tracking and facial ex-pression recognition. Facial activity analysis contains facial feature points tracking, AU recognition and six prototypical facial expression recognition. Current research usual-ly only focus on one or two levels of facial activities, and the information flow is only Bottom-Up, ignoring the dependencies among different levels of facial activities, as well as the Top-Down information flow. To address this problem, we proposed a DBN based model for simultaneous facial feature tracking and facial expression recognition. The pro-posed method systematically, and coherently represent the facial evolvement in different levels, their interactions and their observations. We introduce AU combination node to model the AU correlations, which can significantly reduces the computation cost. Based on the conditional independent encoded in the DBN, learning the DBN model locally requires much less training data. Given the image observation, we recognize three level facial activities simultaneously through probabilistic inference based on DBN. Compared to previous works, the information flow of the proposed model is not only Bottom-Up, but also Top-Down. Hence, not only the facial feature tracking contribute facial expression recognition, but also the expression recognition help improve the tracking performance.We proposed a domain knowledge driven prior model learning method. Prior model based methods for facial activity recognition can effectively handle the image observation uncertain problem. However, the use of prior models faces a bottleneck:Learning the model often requires a large amount of reliable and representative training data, hence the model learnt cannot generalize well to different data sets. To solve this problem, we proposed a knowledge driven prior model learning method. Based on the study of the FACS, an empirical analysis of facial anatomy, and the previous studies, we summarized and extracted the domain knowledge in the facial activity field, which is further transferred to constraints on individual AU, constraints on AU group and constraints on AU dynamic. A fast and effective rejection sampling algorithm is proposed to incorporate the prior knowledge into the DBN model learning process. Experimental results show that the generalization ability of the proposed method is significantly better than data driven prior model.