Multi-Pose Facial Landmark Detection Via Fusing-Feature-Based Convolutional Neural Network

Facial landmark detection is an important research topic in the fields of computer vision,pattern recognition and human-machine interaction,and has wide applications in facial recognition,facial expression analysis and human face reconstruction.Nowadays deep learning based landmark detection approaches become more and more popular,which mainly consist of two steps: 1)first learn nonlinear features which indicate semantic information of faces,using shallow-to-deep neural networks;2)and then map the features to the location of facial landmarks using linear/nonlinear regression learned from training set.Compared with traditional statistic-based detection approaches(such as Active Shape Models,Active Appearance Models),although traditional deep learning based approaches improve both detection precision and efficiency,they still have two issues.First,while human faces have various poses and each pose imposes different feature space,traditional deep learning based approaches do not exploit different poses individually but apply face alignment for preprocessing,which limit the detection precision.Secondly,while different landmarks possess different feature distributions(e.g.,silhouette landmarks and inner organ landmarks such as eyes,nose and mouse),traditional approaches do not treat them individually which leads to non-uniform precisions of different landmarks.Many experiments indicate that facial landmark detection is not an isolated task but closely related to many attributes of faces.Based on the aforementioned argument,we propose a Gray-Edge-HOG(GEH)feature based convolutional neural network,and a pose auxiliary task assisted convolutional neural network for facial landmark detection.The main contribution of the thesis is summarized as follows.(1)We propose Gray-Edge-HOG(GEH)feature based convolutional neural network(GEH-CNN)for facial landmark detection.GEH-CNN first extracts the edge feature,histogram-of-gradient feature and the intensity feature,and then concatenates them into a three-channel space,and finally uses it for input of a CNN and learns both the feature vector and the regression mapping.Experimental results show that,compared with traditional convolutional neural network models,GEH-CNN produces higher precision of detecting outer silhouette landmarks.(2)We propose a pose auxiliary task assisted convolutional neural network(PATCNN)for facial landmark detection.By considering the relationship between facial poses and facial landmark locations,PATCNN learns landmark locations and simultaneously learns facial poses in a joint-task convolutional neural network model,which possesses the feature space of landmarks involving both locations and poses.Experimental results show that PATCNN outperforms traditional convolutional neural network models for detecting landmarks of faces of various poses.