Research On Multimodal Emotion Recognition Based On Spatiotemporal Feature Fusion

In recent years,more and more scholars are focusing on emotional computing,which is dedicated to developing computing systems that can sense,understand and simulate human emotional states in order to bridge the communication gap between humans and machines.Emotion recognition is a key component of affective computing,and the electroencephalogram(EEG)can directly reflect the changing state of emotion and has a strong objectivity,so research based on EEG signals has been widely carried out,but there are some problems: EEG signals are collected through different electrodes on the electrode cap,each electrode has spatial information,and research on this aspect is not fully explored.And most of the studies are limited to the same modal data and ignore the complementarity of different modal features.Based on this,the dissertation has done two works.Firstly,A single-mode emotion recognition model based on CCA feature fusion is proposed.Considering the spatial relationship of EEG signals,a two-dimensional map is formed based on the electrode positions of electrode caps to spatially map the DE features,and the spatial information is learned through a Convolutional Neural Network(CNN).On the other hand,since EEG signals are time-series data,the Long Short-Term Memory(LSTM)network is chosen to tap its temporal dependence.In order to maximize the correlation between spatial and temporal features,Canonical Correlation Analysis(CCA)is introduced to extract the most relevant features.Comparing the emotion recognition accuracy of four models,CNN,LSTM,CNN-LSTM and CNN-LSTM-CCA,the results show that the combined model outperforms the single model and the correlation between features can improve the accuracy of emotion recognition.Secondly,the improved DCCA multimodal emotion recognition model is proposed.On the basis of Work 1,the dissertation further consider the fusion of EEG features and eye-movement features,embed the extraction process of EEG spatial and temporal features by CNN and LSTM respectively into the deep typical correlation analysis model,and introduce eye-movement features,consider the correlation of the two model output features,and construct an improved Deep Canonical Correlation Analysis(IDCCA)model.The model averages the two modal features with the highest correlation and feeds them into a support vector machine for classification.The final comparison of single-modal and multimodal emotion recognition accuracy and confusion matrix under different modalities shows that the model taps into the complementary information between different modal data and has good recognition results.