Facial Expression Analysis Based On Salient Features And Graph Convolution

With the development of human society,images and videos have become more and more important information carriers in our daily life.The understanding of images and videos has become an important technical direction.Automatically analyzing the emotional state of the characters in videos has great value in many practical applications such as family escort,online education,intelligent marketing,and fatigue detection.The expression analysis mainly includes detecting the face in the image,performing static expression analysis of a single frame according to the detected face image,and performing dynamic expression analysis by combining the multi-frame feature.(1)In the face detection process,we propose a face detection algorithm based on salient features constraint,which effectively enhances the network's feature expression ability,while the method does not introduce additional time during the inference phase.The algorithm was tested on the face detection dataset Widerface and achieved good improvement.(2)The static expression classification algorithm under salient features constraint is proposed for the six common facial expression classification task.The proposed method uses the deconvolution network to extract salient features,and uses the facial expression coding system to construct supervised signals of salient features,and combines the loss function generated by salient features with the loss function of the classification task to optimize the network under the multi-task framework.The proposed method has achieved improvment for six common facial expression classification task on the CK+ dataset.(3)Further,we extend the above-mentioned salient features constraint algorithm to a more general expression task.A method of first performing metric learning to obtain a pre-training model and then fine-tuning the task on the pre-training model is proposed.In the process of metric learning,we propose a method of automatically mining saliency features supervision signal by using the triplet,which does not depend on the facial expression coding system.The algorithm was tested on the UNBC McMaster Shoulder Pain facial pain estimation dataset and achieved better results than previous works.(4)Finally,in the video-based dynamic expression recognition task,we propose a a spatiotemporal convolution algorithm which combines local convolution parameter sharing.The algorithm takes the spatio-temporal graph structure constructed by the face feature points in the video sequence as input.The graph combines the position coordinates of the feature points and the feature vectors of the feature points in salient features.At the same time,we use the locally connected graph convolution to optimize the spatio-temporal graph convolution network,which improves the feature extraction ability of the model.The algorithm was tested on Oulu-CASIA dataset and achieved better results than previous works.