Study On Detecting Of Facial Landmarks

The research of this dissertation is face to the demand of face image analysis under uncon-trolled imaging conditions, in the application of service robot. The key point for face image analy-sis is locating a set of facial landmarks in the face image, which is of fundamental importance in the applications like face recognition, expression recognition, gender identification, age identification, face animation, video compression etc.. In the recent decades, facial landmarks detection has been widely researched, and numerous of detecting methods have been proposed. Fairy high detecting rate and locating accuracy have been achieved on the near frontal face images taken in the controlled scenarios. However, it is still open to locate facial landmarks on images taken in the uncontrolled scenarios, with great variations of imaging conditions, such as illumination, expression head pose and partial occlusion due to hair, makeup or decorations.There are two main difficulties for detecting facial landmarks under uncontrolled conditions. First, the variation of imaging conditions brings great changes on the appearance around the facial landmarks, which makes it difficult to model the appearance, especially when the distribution of facial landmarks appearance is highly nonlinear in the appearance space due to imaging condition variance. Second, the ambiguity of local image patches may result to error detection, especially under complicate background (which means there may be locations with similar local appearance). According to the difficulties, this dissertation studies the problem of facial landmarks detection under uncontrolled conditions to provide a solid foundation for the applications of face recognition, expression recognition etc., and improve the intelligence of service robot. We especially focus on the modeling of face local appearance, face shape and the combination of them. There are three main contributions of this dissertation:1. Real-time facial landmark detection method based on cascaded regressionConsider the real-time requirements of many face image analysis applications, we proposed a facial landmarks detecting method based on cascaded regression. First, the Geometry Blurred Pose Indexed features are designed to describe the local image patch around the sample point to accommodate the variations of head pose and expression. Then a series of weak regressors are cascaded to predict the relative offset of the initial pose to the ground truth, with the consideration of scale and orientation. The weak regressors are trained based on the GBPI features with random fern. Finally, a two level cascading framework is adopted to improve the stability and effectiveness of regression. The estimation of a facial landmark pose is started with a randomly selected pose, which is progressively refined by the weak regressors cascaded. Several initial poses are randomly sampled around the mean face shape to generate several estimations, and the final result is calculated based on Mean-Shift. The variation of all estimated locations are used to calculate the confidence of the facial landmarks, and decide if the facial landmark is occluded. The experimental results shows the detecting rate and location accuracy is close to the methods with the considerations of face shape constrains, and even close to the result of human labeling.2. Facial landmarks detecting method based on Markov random field and continues latent variable modelsConsidering the importance of face shape constraints, the distribution of face shape is mod-eled based on probability principle component analysis which serves as the prior probability, and the observation probability models of facial landmarks are designed based on the candidate facial landmarks from the results of cascaded regression, then both of the prior and observation prob-ability are combined under the framework of Bayesian Inference. By take the parameter of face shape model as the continues latent variable, the posterior probability of the facial landmarks given the candidates are estimated and maximized to locate the facial landmarks. Our method is more robust to the face shapes which is not presented in the training set. Considering the existence of local minimum, we model the geometry constrains as a Markov random field and estimate an initial value based on believe propagation. Further considering the variance of head pose, the face shape samples are clustered with k-means, and each cluster is modeled with a continues latent variable model. The models are selected based on the result of Markov random field. The experimental results on the LFPW and LFW database show that comparing to the previous methods, our method improved the detecting rate and locating accuracy.3. Facial landmarks detecting method based on Gaussian mixture models Considering the nonlinear distribution of face shape and local appearance under variation of illumination, pose, expression etc., we model the face shape and local appearance based on Gaussian mixture models (GMMs). For each facial feature point, an SVM with probability output is trained based on the local textures, and then applied to scan a predefined region of the input face image to generate a confidence map which indicates the probability of the occurrence of the corresponding facial feature point.The confidence maps generated are noisy and full of local extremes, GMMs are used to give an analytical approximation of each confidence map based on re-sampling to reduce the noise. GMMs are also utilized to model the distribution of face shape which is highly non-linear due to the variations of identity, head pose and expression. The GMMs of the confidence maps and face shape, which serve as the observation and prior respectively, are combined to calculate the posterior. The posterior is maximized subject to the face shape by iteratively maximizing a lower bound of it. Further more, a Model based Hough Voting Method is proposed to find an initial face shape for initialization。The experimental results on the LFPW and LFW database show that the detecting rate the locating accuracy exceed the result of human labeling.The research of this dissertation improves the robustness, detecting rate and locating accuracy of facial landmarks detecting under uncontrolled conditions, and may promote the application of face image analysis systems under everyday environment.