Facial Expression Recognition Methods Based On Multiple Feature Fusion

Facial expressions are the most powerful,natural,and most common nonverbal means for humans to convey their emotional states and intentions during communica-tion.Facial expression recognition has wide application prospects in many occasions involving human-computer interaction,such as game entertainment,medical rehabilitation,traffic safety,teaching evaluation,product marketing and so on.Due to the complexity and variability of facial expressions,facial expression recognition has become a very challenging research topic.The problems to be solved in facial expression recognition mainly include the following aspects:(1)Most existing facial expression recognition algorithms are easily affected by individual identity factors and lack the ability to cope with identity changes.(2)Most facial expression recognition algorithms based on global features are easy to ignore the local detail features that are crucial to facial expression recognition.(3)The facial expression recognition algorithm based on local features has high requirements on data labels,and the algorithm has high calculation complexity and difficulty in modeling.(4)A single feature based method,whether it is a method based on local features or a method based on global features,when facing complex scenes,the algorithm will have problems such as large performance fluctuations and poor robustness.In view of the above problems,this paper aims to improve the accuracy and ro-bustness of facial expression recognition algorithms,and conducts an in-depth study on the multi-feature fusion learning problem based on static facial expression pictures,and proposes a series of targeted faces Expression recognition algorithm.The main contributions can be summarized into the following four components:(1)Aiming at the problems of large intra-class differences and small inter-class differences of expression features,and ordinary convolutional neural networks are back-ward in extracting local features,a facial expression recognition method based on facial expression bilinear encoding model is proposed.The model uses two feature extractors based on convolutional neural networks to extract features,and uses a bilinear encoding model to combine its outputs to generate feature expressions that highlight local detail features.Experimental results show that the method can effectively extract local features in an end-to-end manner without any local labels,and significantly improve the recognition accuracy.(2)Aiming at the problem that existing methods are difficult to cope with iden-tity changes and cannot fully utilize identity features,a facial expression recognition method based on identity-expression dual branch network is proposed.Unlike previous methods that suppress identity features,this method enhances the influence of identity features and uses a bilinear model to fuse expression and identity features to adaptively learn the relationship between identity and expression.The experimental results show that the new features formed after the integration of identity features are more discriminative,enhance the robustness of the facial expression recognition model to the identity change,and improve the facial expression recognition accuracy.(3)Aiming at the problems that the way of extracting local features in the existing methods are too complicated and the local-global complementary effect is not effectively used,a facial expression recognition method combining global features and local features of the region-of-interest is proposed.Firstly,an attention map generator is designed to obtain a set of attention maps indicating regions-of-interest under weak supervision.Secondly,a bilinear attention pooling is used to generate and refine local features,and a selective feature unit is designed,which allows adaptive weighted fu-sion of global and local features before classification.In addition,the local contrastive loss and global contrastive loss are defined and used to improve inter-class dispersion and intra-class compactness at different granularities.Experimental results show that this method can accurately locate the region-of-interest.Compared with using a single global or local feature,it can significantly improve the recognition accuracy.(4)Aiming at the problems that most existing methods need the assist of additional information to extract local features,and the granularity of the extracted local features is relatively single,a facial expression recognition method using multi-granular fea-tures based on a refined horizontal pyramid network is proposed.Firstly,a horizontal pyramid network is designed,the local features uniformly divided on the feature map are used for classification under different horizontal pyramid scales.It effectively uses the distinguishing ability of each part of the face.Secondly,a refinement mechanism is added to the horizontal pyramid network to improve the outliers generated in the local region due to the uniform division,so that the refined local region has stronger feature consistency.Experimental results on multiple datasets show that the refined horizon-tal pyramid network can greatly improve the recognition accuracy without using any additional local supervision information.