| Facial expression recognition is an important part of face recognition,which has become a research hotspot in the field of human-computer interaction,widely used in fatigue driving,online teaching,lie detection,entertainment and other industries.Facial expression recognition,which involves psychology,biology,statistics,computer science and other disciplines,is a very novel and valuable research direction.At present,the collection of facial expression recognition data gradually changes from the laboratory to the real scene(affected by the mixed interference of lighting,occlusion,posture and other factors),which leads to the difficulty of facial expression recognition In this context,we aim to build and train a neural network model suitable for the real facial expression visual frequency data to improve the accuracy and practicability of facial expression recognition The work content is as follows(1)Considering that the convolutional neural network(CNN)is helpful to extract the invariant features of data,and the recurrent neural network(RNN)is helpful to analyze the time information between consecutive sequence data,inspired by this,this paper constructs an end-to-end CNN-LSTM network model to recognize facial expression video data.First of all,the CNN part uses the classic vgg-16 convolution network to extract the facial expression features of each frame,and the RNN part uses the long short term model network(long short term Memory(LSTM)is used to analyze the differences of expression changes between frames.Secondly,in view of the problem that the weight of CNN cannot be updated by back propagation when CNN and LSTM are trained separately,the end-to-end CNN-LSTM network training is carried out in this paper to make full use of frame invariant features and inter frame related information.Finally,double layer LSTM network and cross layer connection between LSTM layers are used to ensure the effective transmission of deep network features,reducing the risk of gradient disappearance.Experiments on the dataset AFEW and CK+show that the end-to-end network training and optimization of LSTM module effectively improve the accuracy of video expression recognition(2)In view of the complexity and variety of facial expression in real data,neural network is needed to extract more accurate facial expression feature information.In this paper,based on the end-to-end CNN-LSTM network model,the CNN module is improved,and a feature enhanced CNN-LSTM network model is proposed.Firstly,a feature extraction branch is introduced in the middle layer of the main CNN module,and it is integrated with the deep features of the main CNN module to obtain more abundant facial expression details;secondly,in order to further compensate for the loss of local texture features of the image,LBP features are added in the full connection layer of the main CNN module to further enrich facial expression features.Experiments are carried out on two static datasets of FER2013 and SFEW and two dynamic datasets of AFEW and CK+,respectively.The results show that the fusion of different network layer features in CNN model is helpful to describe facial expression features more accurately and improve the accuracy of expression recognition.(3)Considering the high computational complexity of the end-to-end CNN-LSTM network module,the model requires high experimental hardware and is prone to over fitting during training To solve this problem,this paper uses the global average pooling(GAP)layer and self attention(SA)network with lower complexity to replace the full connection layer and LSTM module of CNN module respectively,and proposes a network model based on CNN-SA.First,the sequence image is input into CNN-GAP network for feature extraction to obtain multiple sets of feature vectors;then,the self attention module obtains the attention weight by calculating the correlation between features,and outputs the new feature vector group after weighting the features according to the attention weight;finally,the expression probability values are calculated through the network of full connection layer,activation function layer,dropout layer,etc.Experiments on AFEW data set and CK+data set show that the recognition rate of CNN-SA network is similar to that of feature enhanced CNN-LSTM while reducing the complexity. |
