Affective Analysis Enhanced By Privileged Information And Adversarial Learning

Affective analysis is one of the most important fields of human-computer interac-tion,which includes not only recognizing facial expression directly,but also analyzing multimedia data browsed by users,so as to understand users' emotions indirectly.Priv-ileged information is additional information that can be used during training phase but not be available during testing phase.Making use of privileged information can greatly improve the performance of traditional machine learning.This thesis focuses on two issues related to affective analysis:facial expression recognition and image aesthetic assessment.We adopt the learning framework based on privileged information.We propose to use thermal images as privileged information to assist the training process of visible expression recognition,and propose to use attributes as privileged information to assist the training process of aesthetic rating model.Adversarial learning is the main-stream method of unsupervised learning.It can effectively model the distribution of data and has the effect of regularization.We further propose to adopt adversarial learning to explore the inherent relationship between visible and thermal images and between aesthetic score and attributes.In summary,this thesis proposes an affective analysis method enhanced by privileged information and adversarial learning.The details are as follows:1.This thesis proposes a visible expression recognition method based on thermal enhancement in the label space.The proposed method adopts the framework of learning using privileged information and uses thermal images as privileged infor-mation to assist the learning process of visible expression recognition classifier.Specifically,we first learn two deep neural networks for feature extraction from visible and thermal images.Then we use the learned feature representations to train SVM classifiers for expression classification.We jointly refine the DNNs as well as the SVM classifiers by imposing the similar constraint in the label space.Thermal images during training are then exploited to construct better facial rep-resentations and expression classifiers from visible images.Experimental results on the MAHNOB laughter database demonstrate that the proposed method can ef-fectively exploit thermal images' supplementary role for visible facial expression recognition during training.2.This thesis proposes a visible expression recognition method based on thermal enhancement in both feature and label spaces,which is an improvement of the first method.Specifically,we build two DNNs for expression recognition from visible and thermal images and impose the similarity constraint in the label space.We further add a discriminator in the feature space of the visible and thermal networks,and adopt adversarial learning mechanism to learn modality irrelevant features.Thus,thermal images can improve the learning process of the visible network in both feature and label spaces.Experimental results on the MAHNOB laughter database demonstrate the effectiveness of thermal enhancement in both feature and label spaces.3.This thesis proposes an image aesthetic assessment method assisted by attributes.The proposed method adopts the framework of learning using privileged infor-mation and uses attributes as privileged information to assist the learning process of the aesthetic rating model.Specifically,we first build a multi-task network to learn the aesthetic score and attributes simultaneously.After that,we introduce a discriminator to distinguish the predicted aesthetic score and attributes from the ground truth label.Through adversarial learning between the discriminator and rating network,the distribution of the predicted aesthetic score and attributes is forced to converge to the ground truth label distribution.Experimental results on AADB and AVA databases demonstrate the superiority of the proposed method.