Generative Adversarial Network Based Data Driven Facial Expression Recognition

Facial expression is the most natural and effective way to express human emotions,which is also an important agent of Human Computer Interaction(HCI).With the development of Artificial Intelligence(AI),people place a higher requirement on HCI.They hope the computer could be more “personalized”,and could correctly recognize human emotions.In this way,the computer can make corresponding positive feedback to the users.As one of the important tasks to achieve this goal,Facial Expression Recognition(FER)has received widely attentions from researchers in the past few decades.The relevant FER research results have been adopted by numerous applications in distance education,driver monitoring,and public security lie systems.Despite of high application value in various fields,it remains a difficult task for developing robust algorithms to recognize facial expressions in scenarios with challenging factors,such as insufficient training data,large intra-class expression variations,pose variations,faint expressions,and complex background.Therefore,in this paper,we focus on the facial expression recognition tasks under both laboratory conditions and in the wild,and design a Generative Adversarial Network(GAN)based data driven facial expression recognition model,which can solve the above problems through discrete and continuous face image generation,pose-robust feature extraction,multi-task collaborative analysis,and emotion attention transfer learning.The major contributions of the paper can be summarized as follows:(1)Discrete Generated Data based Facial Expression Recognition.In order to solve the problem of low FER recognition rate that caused by insufficient labeled training samples,we propose an end-to-end discrete generated data based FER model.The proposed method cold jointly model different poses and expressions.Through the adversarial training between the generator and discriminator,the identity representation is explicitly disentangled from both expression and pose variations.As a result,the proposed method can automatically generate large scale labeled facial images by adding the corresponding labels.Finally,a deep expression classification model is trained by both the generated and original facial images,which can promote the FER results.(2)Continuous Geometry Guided Pose-invariant Facial Expression Recognition.In order to solve the problem of low FER recognition rate that caused by insufficient training data and large intra-class expression variations,we design a continuous geometry guided pose-invariant FER model.The proposed method could get the shape geometry feature by facial landmarks from different expressions and poses.Then through geometry interpolation,we can get smooth transition feature between different facial landmarks,which can be used as the shape geometry constraint to generate face images with different expressions and poses in a continuous way.Finally,with the enriched training samples,we can train a deep FER model,and promote the final pose-invariant FER results.(3)Unified Pose modeling for Facial Expression Recognition.In order to alleviate the effect of the poses,we design an end-to-end pose-invariant FER model by unified pose modeling.Through using a GAN based model,the proposed method could generate facial images with arbitrary poses,which can then be used to train a Multi-view Convolutional Neural Network(MvCNN),and joint analyze the facial images with different poses.Through learning poseinvariant features,the effect of the pose variations can be alleviated.The proposed method not only can solve the rigid requirement in MvCNN that each input image must have the corresponding face image in different poses,but also can improve the final FER accuracy.(4)A Multi-task Collaborative Model for Robust Facial Expression Recognition.In order to solve the problem of low FER recognition rate that caused by faint expressions,we propose a multi-task collaborative model for robust facial expression recognition.The proposed method use an end-to-end model for both regression and classification tasks on three coherently related tasks: FER,face synthesis,and face alignment.These three tasks can be mutually restrained and promote each each in such a unified model.A large number of experimental results on three different databases show that the proposed method can not only improve the FER accuracy,but also can promote the facial image generation and face alignment results.(5)A Cycle-consistent Adversarial Network for Facial Expression Recognition in the Wild.In order to alleviate the effect of the complex backgrounds,varied illumination,insufficient training samples,and unconstrained facial expressions,we design a cycle-consistent adversarial network for FER in the wild.The proposed method could utilize large-scale unlabeled web facial images and the unsupervised cycle-consistent adversarial network to transfer the labeled laboratory facial images into web facial images.In this way,we can obtain sufficient labeled wild facial images.Finally,the performance of the FER in the wild can be enhanced by harnessing attention transfer strategy and deep learning method.Detailed research in this paper demonstrate that by generating realistic facial images,learning effective expression representations,designing pose-robust facial expression classification model,collaborating multiple coherently related tasks,and leveraging emotion attention transfer strategy,the proposed end-to-end GAN based deep learning model could solve the FER challenges caused by insufficient training samples,large intra-class expression variations,pose variations,complex background,and so on.Extensive experimental results on four standard facial expression datasets(Multi-PIE,BU-3DFE,SFEW,and EmotioNet)demonstrate that the proposed method could promote the final FER results.