Non-Frontal Facial Expression Recognition Based On Generative Adversarial Networks

Facial Expression Recognition(FER)refers to the process of using computer technology to acquire facial expression images,detect facial expression regions,extract facial expression features and classify facial expression features.Because FER research occupies an important position in the field of computer vision and is also a key technology for realizing human-computer interaction,it has attracted extensive attention and in-depth research from domestic and foreign scholars.At present,the development of FER technology has encountered two key difficulties:(1)the deflection of the head pose in the face image causes facial registration errors and face occlusion,making it difficult to obtain effective facial expression information;(2)lack of sufficient training data causes over-fitting during training.This paper proposes an end-to-end deep learning framework based on Generative Adversarial Networks(GAN).In our proposed framework,a dendritic convolutional neural network model based on ShuffleNet-v2 is introduced to estimate the facial pose in images.Then,based on the frame,different poses and expressions are used to simultaneously synthesize face images and recognize facial expressions in different poses.The method proposed in this paper can effectively solve the problems encountered in non-frontal FER,and has certain practicality.The main work and innovations of this article are as follows:(1)In order to estimate facial poses in non-frontal FER,we propose a dendritic convolutional neural network model based on ShuffleNet-v2.The model uses a single convolutional neural network to model the dendritic structure of facial landmarks.The model is based on the ShuffleNet-v2 network architecture,so it can maintain better performance with fewer parameters.In order to prove the effectiveness of the method,we conducted a large number of experiments on the AFLW,AFW and 300W datasets.Through the four aspects of normalized mean error of facial landmarks,estimation error of facial pose,accuracy rate and recall rate,and compared with existing methods,the results show that the algorithm proposed in this paper has higher accuracy in facial landmarks position,facial pose estimation error is lower,and it has better accuracy rate and recall rate.(2)In order to improve the accuracy of facial pose estimation by the dendritic convolutional neural network model based on ShuffleNet-v2,we propose a facial landmarks localization model based on explicit facial pose estimation for modular auxiliary tasks.An auxiliary model can effectively perform occlusion detection and fine-grained positioning on the facial landmarks on the image.(3)In order to solve the problem of over-fitting in the training process due to lack of sufficient training data,we proposed a FER model that joints poses and expressions based on GAN.To simultaneously synthesize face images and recognize facial expressions in different poses.In our proposed model,the generator based on the encoder-decoder structure can learn the identity features of face images,and automatically generate face images with different poses and expressions with labels,enriching the experimental training set,and the discriminator in the model is processed in mini-batch to directly calculate the statistical characteristics of mini-batch samples,which simplifies the calculation method and improves the stability of the model during the training process.In order to prove the effectiveness of the proposed method,we conducted a large number of experiments on three different datasets Multi-PIE,BU-3DFE and SFEW,and compared with the current advanced FER algorithm.The experimental results show that compared with existing FER methods,our model can not only expand the training set and generate recognition accuracy by generating face images with different poses and expressions,but also the model is more stable during the training process,and good results have also been achieved with extreme poses and face occlusion.