Research On Dynamic Facial Expression Recognition Algorithm Based On Spatiotemporal Multi-feature Fusion

Emotion recognition through facial expression is regarded as one of the most effective methods to directly reflect a person's inner emotional state for affective computing.However,due to the change of facial expression is a dynamic process from subtle to peak to subtle,and a single expression feature has limitations,the dynamic facial expression recognition(FER)based on multi-feature fusion is more popular among researchers.Consequently,a key issue of dynamic facial expression recognition is how to design and fuse features from videos rapidly and thus extract representative spatial and temporal information to improve the recognition accuracy efficaciously.In this thesis,we mainly study the expression recognition based on video from two aspects of various types of feature extraction and feature fusion.The main work of this thesis includes:(1)We first present a new algorithm,the Improved Local Binary Pattern from Three Orthogonal Planes(I-LBP-TOP),which analyzes the contribution of LBP histogram components of three orthogonal planes in the original LBP-TOP algorithm and replaces the lower contribution components with new components.The proposed I-LBP-TOP algorithm can remedy the shortcomings of original LBP-TOP algorithm in feature extraction and can extract both the static and dynamic features in changing expressions.Experimental results on CK+and Oulu-CASIA databases show that the recognition accuracy of I-LBP-TOP is 3.56%and 5%higher than that of the original LBP-TOP,respectively.Besides,the proposed I-LBP-TOP algorithm is superior to the original LBP-TOP algorithm in terms of calculation speed and recognition accuracy on different classifiers.In addition,compared with the other two improved LBP-TOP algorithms(spatiotemporal LBP+Gabor algorithm and LGBP-TOP algorithm),the recognition accuracy of I-LBP-TOP algorithm is also increased.(2)A new representation method of geometric feature is proposed,which takes the facial key points of the parts highly related to expression as image geometric features to obtain the relative position information of expression changes.Further,the spatiotemporal geometric feature is obtained by extending image geometric feature to time dimension.Experimental results on CK+and Oulu-CASIA databases show that compared with the existing geometric feature extraction methods,our proposed spatiotemporal geometric feature can meet the requirements of high recognition accuracy.Moreover,the recognition accuracy of spatiotemporal geometric feature extraction algorithm is better than the existing geometric feature extraction algorithms,and the average recognition accuracy on CK+ and Oulu-CASIA databases is improved by 1.98%?4.22%and 0.41%?4.21%,respectively.(3)A framework that integrates the spatiotemporal motion features(I-LBP-TOP histogram feature and spatiotemporal geometric feature)with static texture feature(Gabor magnitude feature)is designed,which takes into account geometry-appearance and dynamic-still information simultaneously.A support vector machine(SVM)with multiple kernels is applied to train three base classifiers.Finally,to break through the limitation of single feature and realize multi-feature fusion effectively,a decision-level feature fusion method based on a relative majority voting(MV)strategy is employed.The experimental results on CK+and Oulu-CASIA databases demonstrate that feature fusion method based on decision-level can achieve better results than single feature.Compared with the other existing state-of-the-art hand-crafted approaches and some deep neural network algorithms,our proposed method still achieves an improved performance.Moreover,the average recognition accuracy on CK+and Oulu-CASIA databases is improved by 0.75%?8.41%and 1.05%?5.83%,respectively.