Emotion Recognition Based On Privileged Information

As the development of technology and increasing need for personalized commu-nication in human-computer interaction, emotion recognition, which is considered to be a key factor which can influence interaction quality, has been a research hotspot and achieved several progresses. Emotion recognition is usually solved as a machine learn-ing problem. Classical approaches usually focus on feature extraction and classifier design. However, privileged-information-based methods can provide extra knowledge in training phase in addition to features and teaching signals. Privileged information is only available in training, not in testing, which could help to construct better classifiers. Lots of privileged information exist in emotion recognition, for example, in video emo-tional tagging, users’electroencephalogram (EEG) features can be used as privileged information; in EEG-based emotion recognition, video stimulus and users’personalized information can be privileged information; in multiple facial action unit recognition, re-lationships in the same feature space among tasks can be privileged information. This thesis proposes to recognize emotion based on privileged information as follows:1) We propose novel hybrid approaches to annotate videos in valence and arousal s-paces by using both users’EEG signals and video content. First, several audio and visual features are extracted from video clips, and five frequency features are extract-ed from each channel of the EEG signals. Second, statistical analyses are conducted to explore the relations among emotional tags, EEG and video features. Third, three Bayesian networks are constructed to annotate videos by combining the video and EEG features at independent feature-level fusion, decision-level fusion and depen-dent feature-level fusion. In order to evaluate the effectiveness of our approaches, we designed and conducted the psychophysiological experiment to collect data, includ-ing emotion-induced video clips, users’EEG responses while watching the selected video clips and emotional video tags collected through participants’self-report after watching each clip. The experimental results show that the proposed fusion meth-ods outperform the conventional emotional tagging methods that use either video or EEG features alone in both valence and arousal spaces. Moreover, we can nar-row down the semantic gap between the low-level video features and the high-level users’emotional tags with the help of EEG features.2) We propose a novel approach to recognize emotions with the help of privileged in-formation. Specifically, we recognize audience’s emotion from EEG signals with the help of stimulus videos, and tag videos’emotions with the aid of the EEG sig-nal. Firstly, frequency features are extracted from EEG signals and audio/visual features are extracted from video stimulus. Secondly, features are selected by sta-tistical tests. Thirdly, a pair of new EEG and video feature spaces are constructed simultaneously using Canonical Correlation Analysis. Finally, two Support Vector Machines (SVM) are trained on the new EEG and video feature spaces respective-ly. During emotion recognition from EEG, only EEG signals are available, and the SVM classifier obtained on EEG feature space is used; while for emotion video tag-ging, only video clips are available, and the SVM classifier constructed on video feature space is adopted. Experiments of EEG-based emotion recognition and emo-tion video tagging are conducted on three benchmark databases, demonstrating that video content, as the context, can improve the emotion recognition from EEG signals and EEG signals available during training can enhance emotion video tagging.3) We propose a novel emotion recognition approach using subjects and clusters as privileged information. First, five frequency features are extracted from each chan-nel of the EEG signals, and features are selected by statistical tests. Then, we propose two three-node Bayesian networks to capture the joint probability distribution func-tion of emotion labels, EEG features, and subjects or groups during training. During testing, emotion labels can be estimated from EEG features only by marginalized over the privileged information, i.e. subjects or groups. Experimental results on three benchmark databases, i.e. MAHNOB-HCI, DEAP and USTC-ERVS, demon-strate that our approach incorporating subjects and clusters achieves better emotion recognition performance than training classifier for each single subjects, as well as training a subject-independent on the whole dataset.4) we propose multiple facial action unit recognition by modeling their relations from both features and target labels. First, a multi-task feature learning method is adopted after dividing action unit recognition tasks into several groups, and then learn the shared features for each group. Second, a Bayesian network is used to model the co-existent and mutual-exclusive semantic relations among action units from the target labels of facial images. After that, the learned Bayesian network employs the recognition results of the multi-task learning, and realizes multiple facial action recognition by probabilistic inference. Experiments on the extended Cohn-Kanade database, MMI database and the Denver Intensity of Spontaneous Facial Actions database demonstrate the effectiveness of our approach.