Multimodal Emotion Recognition Based On The Fusion Of EEG And Eye Movement Data

Emotion plays an important role in people’s daily life,which is a complex mental state or process.The research has shown that the fusion of different modalities for emotion recognition can help bring out the complementarity of emotion representation between modalities and thus enhance the performance of emotion recognition.Electroencephalogram(EEG)signal-based emotion recognition,which can effectively reflect the real inner state of a person,has become a common method for studying emotion recognition.However,existing multimodal fusion methods often ignore the mutual mapping relationship between different modalities.Therefore,this paper proposes a deep fusion network based on EEG and eye movement data for emotion recognition.The main research of this paper is as follows.(1)A feed forward encoder-based EEG feature extraction method is proposed for EEG emotion recognition.Firstly,three different EEG features(differential entropy,power spectral density,wavelet entropy)were extracted and found to have the strongest emotion representation ability through comparative analysis.Then the differential entropy features were feature smoothed to remove redundant information,the deep differential entropy features were extracted by using feedforward encoder,and the temporal,frequency and spatial domain information of the differential entropy features were integrated.The method is subsequently validated on the SEED-IV dataset,and the average recognition accuracy of the four classified emotions reaches 75.37%,and the experimental results show that the proposed method in this paper can improve the performance of emotion recognition.(2)An eye movement feature extraction method based on feedforward encoder is proposed to solve the problem that eye movement features are not strong in emotion characterization.Firstly,33 eye movement features containing eye movement information are extracted,and then a feature F-test method based on eye movement data is designed to verify the ability of different eye-movement features to distinguish different emotions,and through screening,25 eye movement features that optimally characterize emotions are selected.Finally,the deep features of eye movements are extracted by using feedforward encoder,and the eye movement features with more emotion characterization ability are obtained by reconstructing the interdependence of eye movement features and the corresponding intensity of features.The method is validated on the SEED-IV dataset,and the accuracy of the four-category emotion recognition reaches 65.87%,which verifies the effectiveness of the proposed method.(3)A multimodal attention network model based on EEG and eye movement data is designed for multimodal emotion recognition.Among them,the interactive attention model is used to learn multimodal complementary information and semantic layer contextual information to obtain multimodal emotion representativeness,and the multiheaded self-attention makes the model focus on discriminative features of emotion classification as well as emotion classification.The experimental results show that the proposed multimodal fusion method in this paper has a recognition accuracy of 92.26%on the SEED-IV four-category emotion dataset,and its performance is better than that of single-modal emotion recognition...