Studies On Facial Emotion Recognition Based On Deep Learning

Facial emotion recognition aims to perceive the emotional tendency conveyed by expressions intelligently.It has significant research value and has become a study hotspot in the field of computer vision.For this reason,in view of the shortcomings of the existing facial emotion recognition models,this thesis conducts in-depth research on expression recognition from the two aspects of images and videos.1.For the static expression recognition of images,considering that the Convolutional Neural Network(CNN)pays insufficient attention to the emotion-related local regions and the classic loss functions fail in handling the intra-class differences of expressions,a Hierarchical Attention based Lightweight Densely-connected CNN(HA-LDCNN)is studied.Firstly,the densely connected convolution module is used for features extraction to obtain a multi-scale rich information flow while controlling the number of parameters.Then,a spatial attention unit with small parameters size is constructed to highlight the emotional salient information in the output features of the dense module.Furthermore,an intra-class distance penalty term is introduced to supervise the training of parameters in combination with the classification loss,to reduce the impact caused by the differences in sample identity characteristics.The proposed model is verified on the two public databases CK+ and RAF-DB,and the experimental results demonstrate the robustness and efficiency of the model.2.For the dynamic expression recognition of videos,considering the large quantities of parameters and the neglect of correlations between different modalities in the existing methods based on multimodality,a Decomposed CNNs based Dual-modality Fusion Model(DCNNs-DFM),which extracts the dynamic changes of expressions efficiently,is studied.The model is mainly composed of an appearance module,an optical flow module,and a dual-modality fusion module.The appearance module takes the RGB image sequence as input.It captures the spatio-temporal features of the facial appearance through a 3D decomposed CNN with low complexity.The optical flow module explores the motion states contained in the apex optical flow image through a 2D decomposed CNN.To further improve the recognition ability of model,the dual-modality fusion module integrates complementary information from different networks at the featurelevel under the principle of modality consistency.The experimental results show that the proposed model achieves ideal results on the databases CK+ and MMI,which prove that the model can effectively recognize the dynamic expression.