Research On Micro Expression Recognition Based On Attention Mechanism And Domain Adaptation

Micro-expressions are a type of facial expression characterized by brief duration,subtle magnitude of muscular changes,and natural expression of emotions,which can reveal an individual’s genuine emotional state.It hold great potential for applications in fields such as forensic interrogation and psychotherapy.However,due to the difficulty in collecting and annotating micro-expression samples,currently available micro-expression datasets have limited sample sizes.Micro-expression recognition models based on traditional methods do not require a high number of training samples,but they are unable to extract deep micro-expression features,while deep learning-based methods have difficulty in training robust deep network models by relying only on the available microexpression samples.In addition,micro-expressions only occur in specific regions of the face,making it difficult for existing micro-expression recognition models to effectively extract local facial features.To address these challenges,this thesis investigates micro-expression recognition based on attention mechanism and domain adaptation,the main research work is as follows:(1)Due to the limited number of micro-expression samples,it is difficult to train a robust and effective deep network model,so this thesis proposes a Micro-Expression Recognition based on Adversarial Domain Adaptation(MER-ADA).The model applies an adversarial domain adaptive approach to the micro-expression recognition task,reducing the difference in feature distribution between normal expressions and micro-expression samples,thereby enabling ordinary expression samples to assist in micro-expression recognition.The training and testing process of MER-ADA model is mainly as follows,firstly,the Eulerian Video Magnification algorithm is used to amplify the facial muscle movements in the micro-expression video sequence.Secondly,the apex frame and the onset frame images of both the normal and micro-expressions are aligned and cropped,and the optical flow between the apex and onset frames is extracted to construct an optical flow map.Then this optical flow map is fed into the MER-ADA model for training.Finally,the trained model is utilized for micro-expression recognition.The experimental results show that the model achieved unweighted F1 scores of 0.7286,0.8210,0.7591,and 0.7455 on the SMIC,CASME II,SAMM,and composite datasets,respectively,with unweighted average recall reaching 0.7267,0.8132,0.7295,and 0.7321.(2)To further enhance the generalization capability of the MER-ADA,this thesis proposes a Dynamic Weighting Gradient Reversal Layer(DW＿GRL).DW＿GRL mainly uses the classification probability of the domain discriminator output to assign different weights to different samples,so that the model gives more attention to samples that are easily discriminated by the domain discriminator.The experimental results showed that by incorporating DW＿GRL into the MER-ADA model,the unweighted F1 scores on the micro-expression datasets SMIC,CASME II,SAMM,and the composite dataset were improved by 0.0088,0.0129,0.0103,and 0.0255,respectively,while the unweighted average recall were improved by 0.0078,0.0198,0.0094,and 0.0148,respectively.(3)Due to the fact that micro-expressions only appear in localized facial regions,the basic model MER-ADA struggles to extract micro-expression features with strong discriminative power from these regions.To address this issue,a global and local feature learning module is designed and added to the base model MER-ADA together with the dynamically weighted gradient inversion layer DW＿GRL,and a Micro expression Recognition based on Attention Mechanism and Domain Adaptation(MER-AMDA)is proposed.In this model,the global and local feature learning module firstly divides the global feature tensor extracted by the feature extraction network into three local feature tensors corresponding to the eye,nose and mouth regions in the face image using the feature division unit.Secondly pools the three local feature tensors with the global feature tensor for global averaging to obtain three local feature vectors and one global feature vector.Then the local attention unit assigns corresponding weights to the local feature vectors according to their contribution to the classification result.Finally the weighted local feature vectors are stitched together with the global feature vectors and fed into the classifier and domain discriminator respectively.Compared to the base model MER-ADA,the MER-AMDA model is able to focus on the key local regions of the face associated with micro-expressions,i.e.the eyes,nose and mouth regions,while suppressing the negative effects of irrelevant regions on micro-expression recognition.The MER-AMDA model ultimately achieved unweighted F1 scores of 0.7653,0.8719,0.8035,and 0.7952,and unweighted average recall of 0.7583,0.8607,0.7810,and 0.7858 on the SMIC,CASME II,SAMM,and composite datasets,respectively.