Research On Facial Expression Recognition Methods Based On Multi-task Convolutional Neural Network And Generative Adversarial Network

Facial expression recognition is an important and popular research topic in computer vision.However,facial expression recognition under real-world environments remains a great challenge,which needs to deal with the variations of facial appearance in the facial images caused by different poses,illumination and partial occlusions.The newly emergent deep learning has made remarkable progress in various fields of artificial intelligence and received significant attention,which strongly promotes the development of computer vision.Therefore,research on deep neural network based facial expression recognition is meaningful and of practical significance.The main works in this thesis are summarized as follows:(1)We propose a facial expression recognition method based on the multi-task convo-lutional neural network.Traditional facial expression recognition methods consider feature learning and classifier training as two separate stepswhich may result in the suboptimal solution.Besides,the insufficient training data and the ineffective objective function may increase the risk of overfitting and degrade the generaliza-tion ability of the model when applying deep leaning to facial expression recogni-tion.Moreoverduring the training,the easy-classified samples usually dominate the majority of the training set,and the hard-classified samples are not paid special attention.Based on the above observations,we propose a novel facial expression recognition method based on the multi-task convolutional neural network.We elab-orately designed a multi-task convolutional neural network,which utilizes a joint loss to learn discriminative features for each facial expression.In this way,our method is able to exploit the relationship among all facial expressions to extract the valu-able information.Besides,we introduce two kinds of dynamic loss weights,which makes the training process concentrate on classifying hard-classified samples and hard-classified expressions.The proposed method achieves the recognition rate of 99.03%on the CK+ database;the recognition rate of 86.25%on the Oulu-CASIA database and the recognition rate of 82.34%on the more challenging MMI database.(2)We propose a novel facial expression recognition method based on the generative adversarial network.Although deep learning has made great progress in image clas-sification,there still exist many problems when deep learning is directly app lied to facial expression recognition,since deep learning is a big data-driven technology and the publicly available facial expression databases typically contain a small number of training data.Moreover,labeling of facial images is tie-consuming and expen-sive.The existing methods attempt to utilize multiple databases to t.rain the model,which may lead to the problem of underfitting and degrade the perfo rmance of the model due to the intrinsic data bias among these databases.In order to address the above problems,we propose a novel facial expression recognition method based on the generative adversarial network.We design a facial expression synthesis genera-tive adversarial network(FESGAN)to generate facial expression images,which are used to augment the training data.In order to better take advantage of the expres-sion information in the synthetic images,the recognition network and the FESGAN are jointly trained to boost each other.Besides,to further alleviate the problems caused by the data bias between the real images and the fake images,we introduce an intra-class loss and a novel real data-guided back-propagation algorithm to optimize the proposed loss.Such a way can not only enhance the intra-class compactness of the features for the input images,but also avoid the performance degradation of the recognition network caused by the interference of the synthetic images.The proposed method achieves the recognition rate of 99.34%on the CK+ database,the recognition rate of 88.13%on the Oulu-CASIA database and the recognition rate of 84.81%on the MMI database.