Research On Facial Landmark Localization Based On Cascaded Pose Regression Under Partial Occlusion

As the basis of important tasks such as face recognition,facial expression analysis,and 3D face reconstruction,the topic of facial landmark localization has received extensive attention from researchers and made great progress,specifically,the cascaded pose regression method has achieved superb landmark localization performance which was comparable to the human beings under constrained conditions(e.g.,favorable illumination,no occlusion,frontal face).However,when the face is blocked by various objects such as sunglasses,masks and hair,the performance of cascaded pose regression degenerates significantly.The cascaded pose regression method relies on good initial shape and expressive shape-indexed features,the facial occlusion not only reduces the accuracy of face detection,which affects the initial facial shape,but also corrupts a part of image features of the face,which introduces noise to the shape-indexed features of the method.To solve the problems mentioned above,this thesis conducted a study from two aspects,i.e.,the initial facial shape and the shape-indexed features of the cascaded pose regression method,to improve the quality of the initial facial shape and reduce the impact of facial occlusions on the shape-indexed features,which effectively improved the landmark localization accuracy of the cascaded pose regression under partial occlusions.The work done by this thesis is summarized as follows:(1)For the problem that the initial localization error of cascaded pose regression method is large due to occlusion,a shape initialization method based on five-point estimation and texture similarity is proposed.In order to reduce the error of the initial shape,this thesis firstly estimates the position of five key points on the face(the center of the eyes,the tip of the nose and the corners of the mouth)based on the MTCNN model.Then,the face is divided into four partial regions(left eye,right eye,nose,and mouth),and shape initialization is performed for each partial region separately,and the degree of occlusion of each partial region is roughly estimated.Finally,the most similar face shape in the training set is selected as the initial shape.The experimental results show that compared with the commonly used random shape initialization method,the initial shape error is reduced from 16.13% to 8.61%,and the landmark localization error is reduced from 8.68% to 6.74%.(2)For the problem that facial occlusion brings noise to the shape-indexed features of the cascaded pose regression,one kind of adaptive shape-indexed features based on the occlusion probabilities of landmarks is proposed.The local texture feature of the occluded landmarks is replaced by that of the occluder object,treating the texture feature of the occluder object as the local texture feature of the landmarks would do harm to the landmark localization performance of the cascaded pose regression.In order to reduce the noise brought by the facial occlusion,this thesis assigns an adaptive weight to the local texture feature of each landmark based on the occlusion probabilities of the landmarks,the bigger the occlusion probability is,the lower the weight would be,thus the cascaded pose regression relies more on the texture feature of the unoccluded landmarks,reducing the noise brought by the facial occlusion.To improve the localization accuracy of occluded landmarks with the aid of unoccluded landmarks,this thesis performs the sparse shape reconstruction of the noisy facial shape with the learnt shape dictionary.The experimental results on the COFW dataset show that,compared to the traditional shape-indexed features which were not weighted by the adaptive weights,the landmark localization error decreases from 6.74% to 6.58%,which further decreases from 6.58% to 6.24% with the sparse shape reconstruction.