Facial Expression Recognition In Video

As a way of conveying information,expressions play an important role in human communication.Research on facial expressions can help people understand human mental activities and emotional states.With the rapid development of deep neural networks in the field of computer vision,facial expression recognition in videos has been used widely.At present,facial expression recognition in images has been extensively researched.However,facial expressions are a dynamic process,and videos can reflect the real situation of facial expression changes better.There are three key problems in facial expression recognition in video: Firstly,the existing research work on facial expression recognition in video is mainly used in commercial fields or competitions.Therefore,related research institutions in this field are mainly large enterprises.There is no public code because their code is used for commercial usage,and the entire network must be built from scratch.Secondly,it is necessary to consider both the extraction of static facial expression features of video frames and the extraction of dynamic time features between frames.Thirdly,the video expression dataset has imbalanced data distribution and small dataset problem.In response to the above problems,this paper proposes a video facial expression recognition model that combines global and local time domains.The model introduces transfer learning algorithm to solve the problems in the dataset.The main work and innovations of this paper are as follows:(1)This paper proposes a video facial expression recognition model that combines global and local time domains.The whole model is divided into three modules: global time domain information extraction,local time domain information extraction and weighted voting.In the global time domain information extraction module,CNN combined with a RNN is used to simultaneously extract the expression features of the video sequence and the global time features.In the local time domain information extraction module,3D CNN is used to simultaneously extract the expression features and local time features of the video frame sequence.In the weighted voting module,weighted voting is performed on the output results of the other two modules to achieve better recognition results.Through experiments on the CK+ dataset and the AFEW dataset,the video facial expression recognition model that combines the global and local time domains proposed in this paper respectively achieves the recognition accuracy of 94.19%and 46.71%.(2)Aiming at the problem of imbalanced data distribution and small dataset in the video facial expression dataset,the transfer learning algorithm is introduced on the basis of combining the global and local time domain video facial expression recognition model.The whole model is divided into three modules: the network-based deep transfer learning module,the mapping-based deep transfer learning module and the weighted voting module.In the network-based deep transfer learning module,the network in the module is pre-trained using the VGGFace2 and Image Net datasets,and the trained network structure and parameters are transferred to the video facial expression recognition problem,and then the target dataset is used to fine-tune the network.In the mappingbased deep transfer learning module,the high-level features that extracted from the source domain and target domain datasets through a three-dimensional convolutional neural network are mapped to a high-dimensional space for adaptation,thereby increasing the transferability of high-level features.Through experiments on the AFEW dataset,adding transfer learning methods to the model can solve the dataset problem of imbalanced data distribution and small dataset,and the recognition accuracy rate on AFEW has increased to 59.11%.