Micro-expression Spotting And Classification Based On Deep Learning

Due to its wide application in various fields such as financial security,clinical diagnosis,public safety and emotional monitoring,automatic micro-expression recognition has attracted more and more attention from researchers in recent years.Micro-expressions are transient,weak and not controlled by the subjective will.Automatic micro-expression recognition includes two steps:micro-expression spotting and micro-expression classification.This paper proposes a micro-expression spotting method and a micro-expression classification method,respectively.(1)In view of the small sample size of micro-expression databases,the deep convolutional neural network was applied to spot facial micro-expressions from videos through transfer learning.Firstly,the pre-trained deep convolutional neural network model was selected;the convolutional layers and pre-trained parameters were reserved.Then the full connected layer and classifier were added after these layers to construct a deep binary classification micro-expression spotting network named MesNet.In order to remove the noisy labels from micro-expression databases that disturb the network training,the concept of transition frames and the adaptive algorithm of transition frames recognition were proposed.The experimental results show that the AUC values of MesNet on CASME II,SMIC-E-HS and CAS(ME)~2 reach 0.9556,0.9338 and 0.7853,respectively.Among them,MesNet achieves state-of-the-art results both on CASME II which is a short video database and CAS(ME)~2 which is a long video database.It shows that the proposed MesNet has the characteristics of high accuracy and wide application range.The results of the transition frame comparison experiment show that removing the transition frames from original videos when constructing the training set can effectively improve the micro-expression spotting performance of MesNet.(2)In order to classify the types of micro-expressions,a micro-expression classification network named MecNet is proposed,which is based on the deep convolutional neural network and transfer learning.In order to improve the accuracy of micro-expressions classification of MecNet on the joint database of CASME II,SMIC and SAMM,Meg Net is proposed to expand the training set.Meg Net is a micro-expression sample generation network based on the auto-encoder.Meg Net used Asian micro-expression samples of CASME II to generate western micro-expression samples.Meg Net designs a convolution module to encode images,a feature map upsampling module based on the sub-pixel convolution to decode images,and a loss function based on the structural similarity of images to optimize the network.The generated western micro-expression samples were added to the training set of MecNet.The experimental results show that using Meg Net to expand training set can effectively improve the accuracy of micro-expression classification of MecNet.Combined with Meg Net,MecNet performs better than the most existing algorithms on the joint database composed of CASME II,SMIC,and SAMM.