Micro-expression Recognition Based On Spatial Temporal Information Fusion

Micro-expressions are the manifestations of human try to suppress or hide his real emotions.It has been used in the fields of criminal investigation,political psychology and clinical medicine.The difficulties of establishing micro-expression samples,sensible to noise,and insufficient acquisition of effective spatial temporal information,seriously restrict the recognition rate of related algorithms.In the case of small samples and noise interference,accurate and effective extraction of spatiotemporal information from image sequences is the key to micro-expression recognition.To solve the problems existing in micro-expression recognition methods that are sensible to noise and insufficient extraction of spatial temporal information.This thesis proposes a micro-expression recognition method based on spatiotemporal feature fusion.The main work of this thesis are as follows:(1)In the preprocessing stage.First,the image is divided into regions of interest(ROI).In order to increase the discriminability of features,a differential energy map is used to find ROI regions where different emotions change in motion,so that an adaptive weight ROI can be calculated for different emotions.(2)In the low level spatiotemporal feature extraction,a binary coding method based on spatiotemporal local cube(LCBP)is proposed.LCBP extends the coding region from a three-dimensional orthogonal plane to a cube for coding.The micro-expression motion direction information LCBP_direction is extracted through the designed 8 masks.On this basis,the amplitude LCBP_amplitudes and spatial information LCBP_3D are added.We obtain the underlying spatiotemporal information while effectively denoising,and has a small feature dimension.(3)The LCBP-STGCN method is proposed to obtain the spatiotemporal fusion features.First,the ROI is used as the graph node position,and the LCBP feature is used as the graph node information to establish the spatiotemporal graph structure.Then use STGCN(Spatial Temporal Graph Convolutional Networks)with the ability to process non-European data to extract information between graph nodes.STGCN includes two-dimensional convolutional modes:Graph Convolutional Network(GCN)and Temporal Convolutional Network(TCN).They are used to capture spatial and temporal information between graph nodes,respectively.By alternate convolution of GCN and TCN,the spatiotemporal characteristics of LCBP-STGCN high-level can be obtained by integrating spatiotemporal information.Finally,the SVM(Support Vector Machine)classifier is connected for micro-expression recognition.Finally,experiments were performed on four spontaneous micro-expression databases SMIC,CASME,CASME2 and SAMM.The results show that compared with the current mainstream manual-based recognition algorithms and deep learning-based recognition algorithms,the proposed algorithm has got a higher recognition rate,and has robustness.In a certain extent,the problems of low recognition rate caused by the sensibility of noise to micro-expressions and the lack of spatial temporal information are solved.