Detection And Recognition Of Micro-expression Based On Spatio-temporal Features Fusion

Facial expressions carry a lot of communication information,even more than words and body.Expressions can be divided into two types according to their duration,macro-expressions and micro-expressions.Macro expressions occur in a larger area of the face and are easy to recognize.Different from macro expressions,micro expressions also have the characteristics of low exercise intensity and short duration,which makes it difficult to suppress and hide micro expressions.Human inner intentions and psychological states can be revealed to a certain extent through detecting and recognizing micro expressions,which are of great significance in public security fields such as criminal interrogation and polygraph detection.In recent years,more and more people pay attention to the research on micro-expression,but due to the characteristics of duration less than 0.5s,involuntary and low intensity,the detection and recognition of micro-expression is still a challenging task.In order to help people understand the characteristics of micro-expression,the Ekman team developed a set of micro-expression training tools(METT).People who receive this training can recognize 7 basic types of micro-expressions,but artificial recognition of micro-expression not only relies heavily on professional experience but also has a low accuracy rate.In terms of automated micro-expression analysis,since the motions caused by micro-expression occur in small local areas of the muscles and blood vessels of the face,it is not easy to capture the small deformations.The performance of the currently proposed automated micro-expression detection and recognition algorithms is still a gap from the actual application.Therefore,in response to the above problems,this paper proposes a macro-and micro-expression detection algorithm and a micro expression recognition algorithm.Among them,one is macro-expression and micro-expression detection based on spatio-temporal feature fusion,and another is micro-expression recognition based on fine-grained hierarchical spatio-temporal feature.The research work includes:(1)In order to solve the problem of distorted facial details caused by image registration and the influence to flow field caused by head shaking in the micro-expression detection algorithm based on dynamic texture features,which leads to unsatisfactory detection performance,a micro-expression detection algorithm based on spatio-temporal feature fusion is proposed.First,we remove the global displacement caused by head movement,and decouple the local displacement vector from the overall facial optical flow field.Then,we extract the optical flow feature of the local motion from the region of interest around the facial landmarks,and filter out the patterns that can represent the micro-expressions.The pattern is composed of two parts,the amplitude trajectory and the angle trajectory,which respectively represent the main intensity and direction of the change.Finally,according to the frame rate of the data base and the induced characteristics of macro expressions and micro expressions,a multi-scale filter is used to improve the algorithm's ability to annotate micro expressions and macro expression clips in long videos.(2)In order to solve the problem that there are much redundant information in the facial area extracted in the micro-expression recognition algorithm based on hierarchical spatio-temporal features,a micro-expression recognition algorithm based on fine-grained hierarchical spatio-temporal features is proposed.First,we extract the spatio-temporal features of each level in the micro-expression video clips,and use the projection matrix to establish the connection between the spatio-temporal features.Next,we select the regions that contribute to the recognition task,and then count the levels with the largest overall contribution.Finally,the intersection of the selected area block under this level and the selected area block of the previous level is performed to remove the spatial redundancy of the hierarchical spatio-temporal features and improve the discrimination of micro-expression features.The proposed algorithm not only has better visual effects,but also has significantly improved accuracy,recall rate and F1-score compared with popular algorithms.In addition,the proposed algorithms have relatively good performance in unbalanced samples.