Research On Fine-grained Feature Learning Method For Spontaneous Micro-expressions Recognition

Micro-expressions(MEs)are involuntary facial expressions,which reveal people's hidden feelings and occur when people try to conceal their true feelings.Different from ordinary facial expressions(macro-expressions),MEs often last from 1/25 to 1/2 second,which are spontaneous,subtle,and rapid facial movements reacting to emotional stimulus.Thus,it is imperceptible to capture and recognize micro-expressions in real-time for the vast majority of people,especially for those who lack professional MER training.Therefore,for better assisting humans to perceive and understand people's MEs,making the computer be more “personalize”(i.e.,the computer could properly perceive and understand the real emotion of human,then act intelligent sensitive and friendly response to human emotion),researchers try to dig into automatically MER in computer science.MER has received wide attention from researchers in the past few decades and many relevant research results have been adopted by numerous applications,such as national security,judicial trial,clinical medicine,teaching evaluation,and business negotiation.Despite those high-quality achievements in MER,it remains a tough task for developing robust algorithms to recognize MEs in scenarios with challenging factors,such as insufficient training data,subtle movements in MEs,inadequately correlated action units,and environmental variations.It is difficult for coarse-grained features such as global facial features and expression-shared features to handle these challenges as the coarse-grained features can hardly learn salient and discriminative expression features.Therefore,in this paper,we focus on the research on finegrained feature learning in MER,i.e.,attention features,expression-specific features,and facial local features by harnessing the attention mechanism in deep learning.We design a fine-grained feature learning model for spontaneous micro-expressions recognition under both laboratory conditions and occlusion conditions,which can solve the above problems through transfer learning with macro-expression database,expression-specific feature learning,multi-task collaborative,and weighted region feature extraction as well as relationship reasoning.The major contributions of the paper can be summarized as follows:(1)Attention Transfer Mechanism based Micro-Expression Recognition.To solve the problem of low MER recognition rate caused by insufficient labeled training samples,we propose an attention transfer mechanism based MER model.The proposed method applies the macroexpression database to train the complex teacher network.Through the feature distilling ability of the teacher-student model,the student network can learn fine-grained attention feature maps which are similar to those learned by the teacher network and highlight the expression-related areas.Furthermore,the feature learning and classification ability of the teacher network can be transferred to the student network to enhance the generalization ability of the student model.Through the attention feature learning and transferring,the feature learned by the student network can obtain strong classification ability and the overfitting of in training a deep network can be relaxed.The experimental results on the composite database of MEGC2019-CD and those subdatabases show that the proposed method can effectively promote UF1 from 1% to 30% when compared with the existed methods.(2)Expression-Specific Feature Learning based Micro-Expression Recognition.To alleviate the effect of subtle movements which cause difficulties in discriminative feature learning,we design an expression-specific feature learning-based MER model.The proposed method could learn fine-grained expression-specific features from a set of expression-shared feature which is extracted by a shallow layer-based deep learning backbone.Through the cooperation of constraint from the attention mechanism and expression-specific detection loss,our method could highlight the micro-expressions categories related characteristics,and boost the discriminative of the learned features,which can reduce the influence of low recognition results caused by the subtle movements in micro-expression.A large number of experiments on the composite micro-expression database of MEGC2019-CD and many single databases such as CASMR II,SAMM,and SMIC,validate that the proposed method can effectively improve the micro-expression recognition results,and obtain the competitive or best results with the state-of-the-art.(3)Simultaneous Action Unit Detection and Relationship Reasoning based Microexpression recognition.To alleviate the effect of sparse and inadequately correlated action units in micro-expression,we propose a simultaneous action unit detection and relationship reasoning based MER model.By integrating the fine-grained features of the action units to a set of global micro-expression level features,the classification ability of the features used for MER is enhanced,and the MER results can constrain the learning of action unit features vice versa.Thus,with the action units feature learning and relational reasoning,the relationship between those sparse action units can be effectively constructed and the discriminative micro-expression features can be obtained.In the HDE task on MEGC2018-CD,compared to the state-of-the-art,this method improves the average WAR and UAR by 11% and 21%,respectively.(4)Region-inspired Relation Reasoning Network for Micro-Expression Recognition under Partial Occlusion.To alleviate the effect of partial occlusion in micro-expression recognition,we design a region-inspired relation reasoning network for micro-expression recognition under Partial Occlusion.To research MER under real-world occlusion,we first construct synthetic occluded micro-expression databases by using various masks for the community.Based on the synthetic occluded databases,the proposed method could extract finegrained weighted feature representations for facial local regions for suppressing the influence of occlusion.Then,by exploiting the progressive interactions among these regions,the model implements a process of aggregating fine-grained local features to the relational features for a robust occluded MER method.Thus,the performance of MER under partial occlusion can be enhanced by harnessing region-inspired fine-grained feature learning and reasoning.Experimental results show that in the HDE and CDE tasks of occluded MEGC2018-CD,the proposed method can improve each performance index by 2%?53% on average compared with other methods.