| Micro-expressions are facial expressions that can reflect the true emotion when peoples try to hide their true emotions.The involuntary nature of micro-expressions decides that recognizing microexpressions has huge application valuables,such as lie detection,medical diagnosis,interrogation,investigation and so on.However,recognizing micro-expressions is difficult due to subtle facial muscle movement and fleeting duration.Recently,in the field of computer vision,affective computing and pattern recognition,increasing researchers use computer to automatically recognize micro-expressions,and proposes a lot of methods to solve this task.Yet,there are still so many problems to be solved in the field of micro-expression recognition.Focusing on two challenging problems in micro-expression recognition including 1)more refined representation of facial muscle movements and 2)improving efficiency for micro-expression recognition,this dissertation researches more compact and discriminative facial movement feature representation and proposes a series of the movement feature representation methods and the movement features learning methods with high efficiency.Thus,the compact and discriminative movement features are extracts to represent facial muscle movements,thereby reducing the feature dimension and computational cost,and improving the efficiency and performance of micro-expression recognition.Specifically,this paper includes four main contributions as follows:1.A feature representation method of local binary pattern from oblique planes is proposed.To analyze the facial muscle movements in oblique directions exited in the micro-expressions,the effectiveness of the movement features from the oblique planes is studied for micro-expression recognition,and the feature representation method of local binary pattern from oblique planes is proposed.Local binary pattern from oblique planes can represent the movement information from the oblique planes in the micro-expression video sequence and is concatenated with the local binary pattern from three orthogonal planes(LBP-TOP)to obtain local binary pattern from five intersection planes that can represent the facial movement information from horizontal,vertical and oblique planes.The experimental results show that except for the movement features from the horizontal and vertical planes,those from oblique planes are discriminative for microexpressions,and local binary pattern from oblique planes can supplement the movement feature information from oblique planes,enhance LBP-TOP and improve the performance of microexpression recognition.2.A low-dimensional feature representation method of histogram-of-single-direction gradient is proposed.To explore and analyze a more refined presentation of movement features,focusing on the movement features in concrete directions from horizontal,vertical and oblique planes,the low-dimensional feature representation method based on histogram-of-single-direction-gradient is proposed to study the effectiveness of movement features in different directions.Histogram-ofsingle-direction-gradient can represent the facial muscle movements in a single direction and is concatenated with LBP-TOP to obtain a local binary pattern-single direction gradient feature.Furthermore,histogram-of-single-direction-gradients in 18 directions are sequentially tested,and the effective directions on each dataset are summarized.The experimental results show that 1)the movement features in different directions have a huge difference in performance,including not only effective movement features but also the one that is disturbing,even deteriorating performance;and 2)histogram-of-single-direction-gradient in optimal direction provides discriminative information,and the corresponding local binary pattern-single direction gradient feature can effectively represent facial muscle movements of micro-expressions to achieve excellent performance.3.A micro-expression recognition method based on kernelized two-group sparse representation is proposed.Focusing on the problem that fusing two features will introduce more redundant and interfering information,a feature selection algorithm based on the kernelized two-group sparse learning model is proposed.On the one had,this algorithm can automatically select more discriminative sub-features from two groups of features before feature fusion,thereby obtaining more compact facial movement features of micro-expressions,improving performance and efficiency;on the other hand,for two groups of features,the kernelized two-group sparse learning model can consider their correlation and learn two groups of weights that can represent their contribution.Furthermore.according to the prior information about the effectiveness of two features,the model can flexibly adjust the sparsity of the two groups of features,which eliminates the model’s preference for a certain group of features.In addition,two types of kernelized twogroup sparse learning models are designed,and the model parameters are optimized by the Alternating Direction Method(ADM).Applying the kernelized two-group sparse learning model to LBP-TOP in multiple regions and histogram-of-single-direction gradient in multiple directions can reduce the feature dimension and feature extraction time,and obtain a more compact movement feature representation.The experimental results show that the kernelized two-group sparse learning model can flexibly adjust the sparsity of the two groups of movement features to overcome the model’s preference for a certain group of features,and can learn more compact and discriminative movement features from two groups of different movement features to improve the performance and efficiency of micro-expression recognition.4.A micro-expression recognition method based on geometric movement graph representation is proposed.Extracting the features from the whole facial image leads the model to waste massive effort dealing with un-related regions.To overcome this problem,taking compact facial landmarks as the input of the model,the contribution of compact geometric movement features from facial landmarks is explored for micro-expression recognition,and based on facial landmarks,a geometric two-stream graph network model is built.First,based on the facial landmarks of the onset,apex and offset frames,a geometric movement graph is constructed;then,a separate structure module is designed to aggregate the spatial and temporal features in the geometric movement graph;finally,based on the separate structure module,a geometric twostream graph network model is built to aggregate low-order coordinate information and highorder semantic information in facial landmarks.To avoid the suboptimal nature of the fixed adjacency matrix defined by researchers,a learnable adjacency matrix is introduced to automatically learn the relationship between nodes.Furthermore,considering the strong correlation between facial landmarks,facial action unit and micro-expressions,an adaptive action unit loss function is proposed to guide the learning process of the model,adaptively constraining the multi-scale features to have a synchronous pattern with action units,thereby introducing facial action unit information in a more reasonable and efficient way.The experimental results show that the compact geometric movement features in the low-dimensional facial landmarks are discriminative for micro-expression recognition,and the adaptive action unit loss function reasonably introduces facial action unit information without increasing model complexity and computational cost.In addition,the proposed method provides a new idea to solve microexpression recognition task,which greatly reduces the computational cost and model complexity while ensuring performance,and promoting real-time micro-expression recognition. |
