| Emotion recognition based on objective physiological signals(such as electroencephalogram,blood pressure,and respiration belt)is becoming a hot research problem in affective computing.EEG signal shows outstanding performance in emotion recognition and hopes to reveal the neural mechanism and principle of people’s emotion state,so it has attracted extensive attention of researchers.However,most of the previous research based on EEG rely on the features of a specific domain(time domain or frequency domain).They ignore complementary emotional information between different domain of EEG signals and different physiological signals.The lack of such complementary information will undoubtedly affect the performance of emotion recognition models.In order to solve the problems,We propose an deep EEGfirst multi-physiological affect(DEMA)framework in this paper,the main contributions of this paper are showed below:(1)In order to extract features containing more emotional information from EEG signals,this paper proposes a deep multi-view convolutional neural network(DMCNN).The feature extraction network is used to learn the independent information and capture the complementary information between different domains of EEG.Then we can obtain multi-view EEG features to effectively reflect the emotion changes of subject.(2)In order to sufficiently fuse the emotional information of multiple physiological signals,this paper proposes an EEG-first multi-physiological signal fusion(EFMF)model for emotion recognition.EFMF is inspired by a relevant research in the neuroscience: EEG signal can affect other physiological signals.We constructed the Affective Influence Matrix(AIM)and conducted a series of experiments to explore the coordination and Influence mechanism.Experiment results prove that EFMF can learn the complementary information of EEG signals from other physiological signals and improve the performance of emotion recognition.(3)The framework proposed in this paper is evaluated on the DEAP dataset using different combinations of physiological signals.Experimental results show that the accuracy of DEMA is 97.55% in valence and 97.61% in arousal,which are better than the existing framework.In addition,ablation experiments were performed to explore the combination of different physiological signals,and the results proved that EEG,blood pressure and respiratory bands were the best combination.In conclusion,DEMA can combine the complementary information between multiple physiological signals and different EEG domains to improve the performance of EEG emotion recognition framework. |
