Emotion Recognition Based On Electroencephalogram And Eye Movement Signals

Affective computing is a kind of computing related to,derived from,or able to exert influence on emotions.Its concept was formally put forward in 2001.Therefore,how to accurately identify people’s emotions has gradually become a hot issue in the research of emotion computing.At present,the signals used for emotion recognition mainly include two categories: subjective behavior signals(including facial expression,speech,eye movement signals,etc.)and objective physiological signals(including EEG signals,ECG signals,etc.).The collection of subjective signals is relatively convenient,but it is susceptible to cultural background;Objective signals can more accurately reflect people’s emotional changes,but the collection method is more complex.In view of the advantages and disadvantages of subjective and objective signals,it is a trend to fuse the subjective and objective signals by multimodal fusion method.The fusion vector can more comprehensively reflect the changes of subjects’ emotional state,so as to obtain a more accurate emotional recognition performance.However,there are still the following problems: 1)the existing research mainly focuses on the selection of features,while ignoring the differences of each feature in different emotion recognition tasks;2)The existing research often ignores that non-significant information is easy to be covered in multimodal fusion,which leads to the loss of emotional information contained in non-significant signals;3)In the process of fusion,the correlation mining of modalities and the rationality of fusion framework design are closely related to the effect of emotion recognition.In order to solve the above problems,EEG and eye movement signals are selected for multimodal emotion analysis based on subjective and objective signal fusion,and a subjective and objective fusion neural network(SOFNN)model for emotion recognition is proposed in this paper.The model can effectively learn the temporal and spatial information of EEG signals and dynamically integrate EEG signals with eye movement signals.Then the emotion classification results are output through the classifier.Specifically,this paper first extracts more abundant significant and nonsignificant information from the original EEG signals through a series of 1-D convolution kernels of different sizes,so as to ensure that they will not be covered in the fusion process.Then,this paper designs a subjective and objective feature fusion model,which adjusts the proportion of the two modal features by dynamically learning the weight vector,so that the model can give full play to the advantages of each modality.In addition,the original EEG signals are cut by a specific length of time window and the two signals are always strictly aligned on the time axis during the feature fusion process,so that the two modalities simultaneously correspond to the emotion state of the subjects.Through the relevant experiments on the public dataset SEED-IV,the effectiveness of SOFNN is proved.For the recognition tasks of happy,sad,fear and neutral,the accuracy of SOFNN is 86.27% and the standard deviation is10.16%,which are better than the existing methods.The experimental results show that our model can make better use of the complementary relationship between subjective and objective features to achieve better emotion recognition performance.