Generative Adversarial Network Based Data Augmentation For Facial Expression Recognition

Facial expression recognition is an important branch in the field of computer vision,which plays a vital role in sociable robots,medical treatment,driver fatigue surveillance and many other human-computer interaction systems.Due to the powerful representation capability of deep neural networks,deep learning-based methods have made remarkable achievements in many computer vision tasks,but their performance relies on large-scale labelled training data.However,the publicly available expression datasets typically contain a small number of training data,so deep learning-based expression recognition is still far from being satisfactory.Generative adversarial networks are able to generate synthetic samples that simulate the distribution of real samples from training data,and have achieved excellent results in the field of data augmentation.Based on this,this thesis uses generative adversarial networks to augment expression data to improve the performance of facial expression recognition.The main work of this thesis is summarized as follows.(1)The identity diversity of synthesized images by existing facial expression data augmentation methods is insufficient.However,identity diversity is directly related to image diversity,and lower identity diversity inhibits the performance of expression recognition.In response to this problem,a facial expression recognition algorithm based on data augmentation with pretrained StyleGAN is proposed.This algorithm uses the StyleGAN pre-trained on a large-scale face dataset to synthesize expression images with new identities,so as to increase the identity diversity of expression datasets and improve the performance of expression recognition.First,expression images are projected into the latent space of StyleGAN by GAN inversion to obtain the latent codes.Second,the identity information learned by StyleGAN from the large-scale face dataset is transferred into expression images by manipulating the latent codes.This algorithm introduces an intra-class loss to reduce the data bias between synthesized images and real expression images,and introduces the label smoothing regularization technology into cross entropy loss to consider the distribution of non-ground truth classes in synthesized images to improve the performance of expression recognition.Finally,experimental results on CK+,RAF-DB and SFEW datasets demonstrate the effectiveness of the algorithm.(2)The synthesized images by existing facial expression data augmentation methods lack the in-the-wild variations caused by unconstrained imaging factors.It is difficult for expression recognition models trained on these synthesized images to obtain good performance in unconstrained environments.In response to this problem,a facial expression recognition algorithm based on unconstrained image synthesis is proposed.This algorithm learns the unconstrained style from a large-scale in-the-wild face dataset,and then transfers it into small expression dataset images to synthesize unconstrained expression images,so as to improve the performance of expression recognition.In the algorithm,the unconstrained image synthesis module is used to synthesize images,and the expression recognition module uses the synthesized images for training.The algorithm combines these two modules in an end-to-end manner.The free parameters in these two modules could be jointly co-adapted and cooperated through the task-specific loss to improve the performance of each other.Finally,experimental results on RAF-DB,Wider Face and ExpW datasets demonstrate the effectiveness of the algorithm.