Multimodal Emotion Recognition Based On Facial Expression Images And EEG Signal

Traditional emotion identification is mainly studied through face images,body movements,speech semantics and other single modes.Although these external behavioral signals can be easily used for emotion identification,they are prone to camouflage problems,and sometimes they can not truly reflect person’s emotional states.EEG signal is neural electrophysiological activity accompanying the processing of emotions in the brain,which can effectively avoid the problem of camouflage and reflect emotions truly and accurately.Aiming at solving the low accuracy of single mode emotion identification,this thesis studies multi-modal fusion method.Decision level fusion algorithm is used to fuse facial expression image mode and EEG signal mode for emotion identification.The main work is as follows:(1)Based on single mode of facial expression images,this thesis explores two kinds of networks: multi-level convolutional neural network based on CNN and deep residual shrinkage network.Aiming at raising the effectiveness of CNN in extracting whole face or partial face features for image recognition,a multi-level convolutional neural network model is designed and studied for facial expressions identification.To alleviate the problem of network overfitting,this thesis adopts some data enhancement operations in the procedure of model training and learning.In DEAP facial expression image dataset,the identification accuracy of titer dimension and arousal dimension,CK+ dataset and Fer2013 dataset reaches 88.64%,87.32%,96.36% and 73.00% respectively.In view of the influence of deep network on network performance,that is,the deeper the network layer,the more likely to cause network degradation,this thesis adopts deep residual shrinkage network,and the validity of the ameliorated network is demonstrated by experiments.In DEAP facial expression image dataset,the identification accuracy of titer dimension and arousal dimension,CK+ dataset and Fer2013 dataset reaches 91.83%,89.28%,96.97% and73.70%,respectively.(2)Based on single mode of EEG,this thesis explores a variety of EEG emotion identification models,and designs the FFT-Bi LSTM-Attention(FFT＿Bi LA)model.First,Fast Fourier transform(FFT)is used to extract EEG features.Considering that EEG features are a kind of time sequence information,it is necessary to combine past and future feature information to predict the most possible current emotion category,and Bi LSTM is adopted for emotion identification.Moreover,an attention mechanism is also joined to the network model to command the distribution of attention weight to make the model focus on important parts and reduce the impact of irrelevant information on the model.The experimental consequences show that this model can validly ameliorate the accuracy of EEG emotion identification.The identification accuracy of DEAP EEG dataset titer dimension,arousal dimension and Kaggle EEG emotion dataset reaches 90.82%,90.69%and 99.10%,respectively.(3)Based on multi-modal fusion of facial expression images and EEG signals,this thesis presents a multi-modal decision-level fusion emotion identification model architecture which integrates deep residual shrinkage network and FFT＿Bi LA network model to carry out multi-modal emotion identification.The decision-level fusion adopts DS evidence theory fusion algorithm,which integrates the consequences of two kinds of network models with its synthesis rules and decision method to obtain the results of multimodal fusion emotion identification.The identification accuracy of titer dimension and arousal dimension in DEAP dataset reaches 95.51% and 95.36%,respectively.The experimental results show that the emotion identification result of the multi-modal fusion architecture is better than that of the single mode.