Micro-Expression Recognition Based On Heterogeneous Data And Multi Feature Fusion

After years of research,macro-expression recognition has made outstanding achievements.In real life,due to the tendency of human beings to hide their natural expressions,emotional expression tends to shrink from macro-expression to micro-expression.Micro-expression is a brief,rapid and spontaneous posture change of facial muscles to express personal real emotions.It has important practical significance in various fields such as national security,criminal investigation,psychopathology,social communication and so on.The early micro-expression recognition system is generally based on traditional hand-crafted features.In recent years,deep learning has been brilliant in the field of computer vision.Due to the short duration and subtle changes of micro-expressions,automated micro-expression recognition is still a highly challenging research topic in the field of computer vision after entering the deep learning era,and extracting subtle and short-lived muscle movement features of micro-expressions on two-dimensional images is a difficult task for deep learning networks.Extracting subtle and transient muscle motion features of micro-expression from two-dimensional images is a difficult task for deep learning network.At present,the research of micro-expression recognition has the following problems:(1)even if the composite-database is used,micro-expression still faces the problem of low recognition accuracy or over fitting caused by lack of data.(2)The database samples have obvious distribution differences,and the classification characteristics of a few samples can not be fully characterized,resulting in the poor classification ability of the model for a few classes and the problem of over fitting the database,which is lack of practicability.(3)The subtle differences between micro-expression database samples make the single feature deep network face the problem of insufficient representation ability.At present,many micro-expression works focus on the facial two-dimensional spatiotemporal features generated by frontal facial micro expression,and there is little work to use the facial three-dimensional spatiotemporal features.In view of the above problems,this paper proposes several ways to alleviate the existing difficulties of micro expression recognition:(1)facing the problem of lack of micro expression data,micro-expression frames that amplified 20% of the total amount of sequence within the apex frame neighbourhood served as the apex frame usage,significantly enhancing the diversity of the data samples.(2)Facing the problem of database sample distribution difference,the first is to use the focal loss loss function to assign appropriate weights to each category to enhance the classification effect of the model.Secondly,channel attention is used to assign appropriate weight in multi stream fusion.Finally,in the evaluation parameters,unweighted average recall rate and unweighted F1 score are used to evaluate the experimental results and strengthen the reliability of recognition accuracy.(3)Facing the problem of poor representation ability of deep model caused by subtle differences of micro-expression,a variety of heterogeneous micro-expression features are extracted from limited training data,including 3D facial spatial features obtained from 3D face reconstruction,optical flow features,and their complementarity is used for fusion to improve the accuracy of micro-expression recognition.In order to realize the above ideas,this paper proposes an end-to-end training heterogeneous deep feature extraction and fusion network model.Specifically,the research contents of this paper are as follows:(1)This paper investigates the current research status of micro-expression recognition,introduces traditional methods based on hand-crafted features and micro-expression recognition methods based on deep neural network,lists some commonly used micro-expression databases,and analyzes the purpose and significance of micro-expression recognition.(2)The basic principles of face 3D reconstruction technology,image convolution neural network and optical flow estimation technology used in this paper are listed.The related algorithms in the field of micro-expression recognition are investigated and compared with this method,and the differences between this method and related algorithms are analyzed.(3)Micro-expression recognition method based on multi-channel deep neural network of heterogeneous data.This paper introduces the multi-channel deep neural network model(HDF-Net)based on heterogeneous data fusion,and expounds the structure and basic principle of each channel.(4)Based on this method,experiments are carried out and the experimental results are analyzed.Firstly,we illustrate the input preprocessing and parameter selection of multi-channel deep neural network in this paper.Then,we conducted sufficient ablation experiments on the proposed HDF-Net,and analyzed the role of each module of the experiment.Finally,we carry out a large number of experiments on CASMEⅡ,SAMM,SMIC and their composite-database to verify the advantages of this method compared with other methods.