The 3D Convolutional Network For Emotion Recognition Based On Multimodal Physiological Signals

Emotion is the foundation of human daily life and plays an important role in perception,decision-making,and interpersonal communication.Emotion recognition research based on physiological signals can deeply understand the emotional information from signals,perceive complex emotional states,and quickly make accurate judgments.The research can be widely used in human-computer interaction,disease diagnosis and other fields.At present,many emotion recognition studies often ignore the topological features of electroencephalography(EEG)electrodes and only focus on the local features of EEG signals,which is difficult to improve the accuracy of the model in identifying emotional states.In addition,the algorithm convergence speed of the existing emotion recognition methods is slow and the complexity is too high,which can easily lead to a long training time.In order to solve the above problems,this paper starts from the spatial correlation of EEG signals and relocates the topological position of EEG electrodes to improve the recognition performance of the model.Then this research adds the eye movement(EM)modal and spatial attention modules to improve the speed up of the running speed of the model.The main research contents of this paper are as follows:(1)The emotion recognition model based on EEG electrode repositioning method and the three-dimensional convolutional neural network(3D-CNN)with different scales of kernels is proposed.This method aims to extract the spatial relationship between adjacent EEG electrodes and explicitly considers the extraction method of spatial association features to achieve the task of emotion recognition.Specifically,the research first designs the two-dimensional topological matrix according to the position of original EEG electrodes and relocates the one-dimensional electrode position to the two-dimensional map to mine the correlation between electrodes.The 3D-CNN based on the different scales of convolution kernels can extract spatiotemporal features from EEG and perform emotion recognition tasks by the spatiotemporal information of EEG.The emotion recognition accuracies reach 95.67% and 89.88% on DEAP and SEED-IV datasets,respectively.Experimental results prove the validity of the spatiotemporal features of EEG and the advantage of emotion recognition of the proposed model.(2)The emotion recognition model based on the EEG and EM feature fusion method and the spatial attention module is proposed.This research can extract and fuse emotional features from EEG and EM by the multimodal feature fusion method and the spatial attention module so that achieve information complementarity of EEG and EM features,so as to make the model more efficient in the recognition process.Firstly,the work extracts the primary EM features and supplements the valid relevant emotional information.Subsequently,the 3D-CNN model based on the spatial attention module can extract the spatiotemporal features and fuse with the EEG features extracted by the first work,which can extract deeper emotional information for emotion recognition tasks.In the emotion recognition experiments on the SEED and SEED-IV datasets,the recognition accuracy of a model decreases.However,this research improves the convergence speed and running speed of the model,which can be applied to the real-time interaction field.