Design And Implementation Of Robust Expression Landmark Localization Based On Deep Learning

With the rapid development of computer science,facial expression recognition technology,as one of important direction of affective computing,has been a hot topic for many scholars.In recent years,the research on deep learning has made a lot of ground-breaking progress,which brought many opportunities of innovation and breakthrough for other fields.In this thesis,we make an intensive study of facial landmark localization algorithm.In consideration of the nonlinear mapping capacity of deep convolutional neural network,we achieve and compare three kinds of facial landmark localization algorithms which are based on different network.Besides,we compare the three method with other traditional facial landmark localization method.In view of the symbiosis of expression landmark in facial action units,we propose a facial conjugate axis detection and intensity estimation algorithm with robust expression based on deep multi-task learning.The research contents include the following three aspects:(1)In order to compare with the landmark localization algorithm based on deep learning,the traditional Active Shape Model(ASM))algorithm and Robust Cascaded Pose Regression algorithm are studied and implemented in this thesis.ASM is a deformable model based on statistics,which build deformable model by training and utilize the parametric update of affine transformation to matching the landmark of local texture model.However,this algorithm does not possess robustness in pose and occlusion.Robust Cascaded Pose Regression algorithm were proposed on the base of Cascaded Pose Regression algorithm.It employs regression model and bring in facial shape index feature as well as occlusion detection,which achieved a good robustness on pose and occlusion.(2)In this thesis,we choose convolutional neural network to complete the work of feature learning.We study and achieve three different kinds of facial landmark localization algorithms with robustness based on deep learning,respectively Deep Convolutional Neural Network(DCNN),Coarse-to-Fine Deep Convolutional Neural Network(CFCNN)and Task-Constrained Deep Convolutional Neural Network(TCDCN).The DCNN algorithm adopt a cascaded structure with three level,which utilize unsupervised learning to train every layer of the network.The next level adjust the prediction result on the basis of the previous level.DCNN can detect five facial landmarks.CFCNN algorithm can localize 68 facial landmarks,which adopt two independent cascaded network to predict 51 inner points and 17 contour points respectively.CFCNN can localize points with high accuracy but the robustness for pose and occlusion is not very well.TCDCN algorithm combine deep learning with multi-task learning,which adopt a simple structure with non-cascaded convolutional neural network.In this approach,our aim is to optimize the main task,which is facial landmark localization.Head pose estimation as the assistant task,we adopt joint learning for the two above.TCDCN improves the robustness for facial poses,which achieves more robust and faster detection for 68 landmarks.From the experimental analysis conducted on AVEC 2012 expression library?self-built database,and LFPW database,we can conclude that,among the five typical algorithms take part in comparison and analyses,TCDCN performs a better detection result in many situations.The detected facial landmarks by TCDCN can be candidate points to describe expression changes.(3)In view of the symbiosis of expression landmark in facial action units,we propose a facial conjugate axis detection and intensity estimation algorithm with robust expression based on deep multi-task learning.Facial action units(AU)are basic units to coding human expression changes and their inner landmarks are commensal when facial expression appear.The intensity of AU becomes the descriptor of psychology index(Arousal?Valence?Expectation and Power)corresponding to expression.First,we utilize TCDCN to localize facial anchors as candidate points to describe expression changes,then we extract facial geometry features and appearance features from candidate points and form the feature descriptor.In this section,we take the symbiosis of facial anchors in AU region as uniform constraint,we utilize support vector machine and support vector regression to achieve robust expression conjugate axis detection and intensity estimation respectively.The experiment processed in SEMAINE and DISFA expression dataset show that the algorithm proposed in our paper can preferably detect and localize expression conjugate axis and estimate the intensity.