Research On Learning Discriminant Features For Micro-expression Recognition

Different from macro-expressions,micro-expressions are spontaneous and unconscious facial movements,which cannot be deliberately suppressed or hidden,and can reveal people’s genuine emotions.Due to the small sample size of public micro-expression databases,applying deep learning methods to micro-expression recognition is prone to the over-fitting problem.In addition,due to the short duration of micro-expressions and the small changes in involved facial muscles,it is difficult for existing micro-expression recognition methods to effectively extract the discriminative features of micro-expressions,resulting in low recognition accuracy.Aiming at the above problems,this thesis studies discriminative feature learning methods for micro-expression recognition.The main research contents are as follows:(1)To reduce the computational complexity of the micro-expression classification task and solve the problem of insufficient samples in the micro-expression databases,a micro-expression recognition method based on the dual-stream convolutional neural network is proposed.Firstly,the micro-expression samples are preprocessed by using the Euler video amplification algorithm with different amplification factors.On the one hand,the sample size is expanded,and on the other hand,the feature differences between different types of micro-expressions are amplified.Secondly,a dualstream convolutional neural network is constructed.Thirdly,the apex frame image of the microexpression video sequence and the optical flow maps(calculated from the onset frame and the apex frame)are used as inputs to train the model.Finally,the trained model is used for micro-expression recognition.This method takes into account the spatial appearance information and temporal optical flow information,and only uses the onset frame and the apex frame of the micro-expression video sequence,which simplifies the calculation of the micro-expression classification task,and also enables the model to effectively learn micro-expression discriminative features.The experimental results show that the accuracy of this micro-expression recognition method on the SMIC and CASME II reaches 67.07% and 72.72%.(2)To solve the problem that the micro-expression recognition model cannot fully and comprehensively learn the discriminative features of each region of the face during the training,a novel method of generating micro-expression samples with the auxiliary label is proposed.For a certain micro-expression apex frame sample,this method selects another apex frame image of the same experimental subject but different from its category,crops the image blocks of the eye and mouth regions,and replaces the corresponding regions of the apex frame image with the cropped image blocks to get a new composite image.The corresponding composite optical flow maps are obtained in the same way.In this way,the generated sample has two labels,one is the auxiliary label to which the eye and mouth image blocks belong,and the other is the original label of the non-eye and mouth regions.The micro-expression samples with the auxiliary label generated in this way are input into the model for double labeling training.During the training,with the help of loss function,the model can be promoted not only to focus on specific regions(mouth and eye regions,etc.)with relatively obvious muscle movements but also to focus on some regions(non-mouth,non-eye regions)with less obvious muscle movements so that it can fully and comprehensively learn the spatiotemporal features of each region of the face and improve the accuracy of micro-expression recognition.The experimental results show that after using this method,the accuracy of the microexpression database SMIC and CASME II is improved by 3.05% and 3.49%.(3)To make the micro-expression recognition model pay more attention to the regions with more discriminative features and solve the problem that the two streams of the dual-stream convolutional neural network extract relatively independent features,the attention mechanism of the class activation map is proposed.The spatial and temporal streams of the dual-stream convolutional neural network based on the attention mechanism of the class activation map are not independent of each other.The spatial stream generates a class activation map and uses this activation map to enhance attention to the input of the temporal stream.The class activation map generated by the spatial stream shows which regions in the apex frame image have more discriminative features.To make the model focus on these discriminative regions in the optical flow maps,the class activation map is used to perform attention enhancement on the input of the temporal stream.The class activation map is the prior knowledge generated by the spatial stream,and the temporal stream has the information supplemented by the prior knowledge to enhance the model’s ability to learn spatiotemporal discriminative features,thereby improving the accuracy of micro-expression recognition.Experiments show that the accuracy of the model that is trained with generated micro-expression samples with the auxiliary label on the SMIC and CASME II reaches 71.95% and 78.63%.