| Emotion refers to a complex subjective psychological experience that people have in response to external or internal stimuli,typically involving a comprehensive experience of sensation,thought,and behavior,and plays a crucial role in daily interpersonal interactions.With the rapid development of computer and human-computer interaction technology,how to accurately identify human emotions to construct more humanized human-computer interaction environments has become a problem of concern to many researchers.Many studies have shown that emotional states are related to electrical activity in the central nervous system.EEG signals have excellent time resolution,especially in directly measuring neuronal activity.These signals cannot be manipulated or simulated to counterfeit emotional states,and are therefore more reliable than other modalities,but decoding this information and mapping it to specific emotions poses considerable challenges.In this paper,we propose an improved end-to-end convolutional neural network model based on cross-subject EEG emotional data,which addresses the issues of low accuracy in cross-subject problems,poor feature extraction,insufficient deep emotional feature mining of EEG signals,and poor model generalization performance in existing EEG-based emotion recognition models.We also propose a dynamic downsampling algorithm to address the problem of large differences in extreme data volume across different experiments while retaining data characteristics and reducing dimensionality.In addition,to validate the effectiveness of the improved end-to-end convolutional neural network model,we propose a multimodal improved end-to-end convolutional neural network model based on crosssubject EEG and eye movement emotional data using a self-attention mechanism.The contributions of this article are summarized as follows:(1)A new method based on an improved end-to-end convolutional neural network(CNN)system using cross-subject emotional EEG data is proposed in this article.First,to perform cross-subject data training,a reconstruction method is proposed based on the actual meaning of EEG data,which integrates the data of 15 subjects in the SEED dataset into one matrix,constructing high-dimensional feature data.By adding different numbers of batch normalization(BN)and dropout layers at different positions in multiple models,the influence of BN and dropout layers on the model is investigated in depth.In addition,based on the r reconstructing the cross-subject data,a novel improved end-to-end CNN model called SACNN is proposed,which differs from the previous space feature dimension based on EEG and focuses on the time dimension of EEG data for emotion extraction.To further improve the accuracy of recognition,this article performs channel selection on 62 channels of EEG and selects the 10 most relevant EEG channel features for emotion,thereby refining the emotion features and improving the representation characteristics of EEG data.The results show that SACNN based on original crosssubject EEG data and multiple-channel data after selection achieves higher accuracy than other models on the SEED dataset,with accuracies of 88.16% and 91.85%,respectively.(2)In order to further validate the effectiveness of the SACNN approach,this paper conducted research on emotion recognition based on both eye movement signals and EEG signals,and investigated in depth the influence of kernel size on the model.As two different modal signals,eye movement and EEG have their own independent emotional feature information,with significant differences in the amount of data in different experiments.In addition,the emotional features extracted from EEG and eye movement have certain redundancies.Therefore,in order to address these two key issues,this paper proposes a downsampling method to solve the difference in data volume and facilitate the construction of cross-subject matrix data.To address the problem of modality fusion of EEG and eye movement,a self-attention mechanism is used to assist the neural network in effectively extracting stronger emotional features from these modal information.Based on SACNN,an improved end-to-end convolutional neural network model for crosssubject multimodal emotion recognition,SCANN-Multi,is proposed.The model achieved an accuracy rate of 82.37% on the SEED-IV dataset,further validating the effectiveness of the SCANN method for cross-subject data. |
