| Face expression is the most direct and effective way for human beings to express emotions.Accurate recognition of face expressions is able to effectively improve man-computer interaction experience,and allow machines to better understand human emotions and psychological states.In recent years,face expression recognition under constrained conditions has developed relatively mature,while face expression recognition under unconstrained conditions is still confronted with many challenges.For example,variations in pose and illumination,glasses,hair and other face occlusions,identity,age,race and other inherent differences,all of them may affect the performance of face expression recognition.Therefore,this paper focuses on face expression recognition under unconstrained conditions and the specific work is summarized as follows:(1)To better extract face expression features and perform recognition,this paper designs a basic network module based on hybrid CNN and Transformer,called MSwinT.The MSwinT Block uses convolution to extract local features,then calculates multi-head self-attention to model global features,finally exports the fusion of local features and original input features.On the basis,this paper proposes a face expression recognition network based on MobileNetV3 and MSwinT,called HMSwinT,which extracts shallow-level face features by using the lightweight convolution network MobileNetV3 and uses MSwinT Block to capture the global dependency between local features and overall features for deep-level expression features extraction.Experiments on two laboratory face expression datasets of CK+and JAFFE,and three real-world face expression datasets of AffectNet,RAF-DB and SFEW demonstrate that HMSwinT model can significantly improve the accuracy of face expression recognition under unconstrained conditions with less model parameters.(2)To reduce the impact of pose variations on face expression recognition,a pose standardization algorithm based on Attention Residual Equivariance Mapping(AREM)is proposed,which maps the profile face to frontal face at deep feature layer.This method first takes the features extracted from HMSwinT model as the initial features,which are further weighted by channel attention and spatial attention successively.The weighted features are then used as residuals to compensate the missing information from the profile face to frontal face.Experiments on three real-world face expression datasets,AffectNet,RAF-DB and SFEW,demonstrate that the proposed pose standardization algorithm can effectively enhance the robustness of face expression recognition systems affected by pose variations,and improve the accuracy of face expression recognition recognition.(3)To reduce the impact of illumination variations on face expression recognition,an illumination standardization algorithm based on scale decomposition and image processing chain is proposed.This method first uses the Logarithmic Total Variation model to decompose the face expression image,then large-scale components and small-scale components are processed respectively.For the large-scale components containing more illumination information,Histogram Equalization,Gamma Correction,Difference of Gaussian filtering and Contrast Equalization are sequentially carried out on it to remove the influence of illumination.For the small-scale components that are not affected by illumination variations,Laplace sharpening is used to enhance its face detailed texture.Finally,the face expression image under normal illumination is obtained by the fusion of the processed large-scale components and small-scale components,which is fed into the HMSwinT model fused with pose standardization module for face expression recognition.Experiments on two laboratory face expression datasets of Extended Yale B and CMU PIE,and three real-world face expression datasets of AffectNet,RAF-DB and SFEW demonstrate that the proposed illumination standardization algorithm can effectively enhance the visualization effect of de-illumination for face,and improve the accuracy of face expression recognition. |
