Cross-subject Multi-view Emotion Recognition Based On Physiological Signals

Emotion recognition based on physiological signals can overcome the shortcomings of being easily disguised,which has attracted more and more attention in recent years.Physiological signals of different views(i.e.,different features or different modalities)carry rich emotional information.In multi-feature cross-subject emotion recognition based on EEG,frequency-domain(FD)features reveal the activated patterns of individual local brain regions responding to different emotions,whereas brain connectivity(BC)features involve the coordination of multiple brain regions for generating emotional responses;in multi-modal emotion recognition,eye movement and EEG respectively represent external subconscious behaviors and internal physiological responses.Therefore,the complementarity between different view data can be used to construct more discriminative and comprehensive emotional features,resulting in improving the accuracy and robustness of emotion recognition.However,the application of cross-subject emotion recognition based on multi-view physiological signals still has some problems:(1)Due to individual differences in EEG signals of different subjects,the performance of the fusion of these two types of features in cross-subject emotion recognition remains to be fully investigated,and the properties of critical frequency bands,complementary properties of different emotion,critical channels and crucial connections need to be further explored.(2)Effectively fusing multiple subjects’ eye movement and EEG modal data is still a challenging task,which requires balancing the consistency and divergences between heterogeneous modalities across multiple subjects,considering intra-modality and inter-modality information across multiple subjects,and overcoming individual differences of multiple subjects.In order to solve these problems,the main work of this thesis is as follows:First,this thesis first attempt to investigate the fused features of frequency domain features and brain connectivity features extracted by EEG for cross-subject emotion recognition from multiple perspectives,including critical frequency bands,complementary characteristics for different emotional states,critical channels,and crucial connections,using a fast and robust approximate empirical kernel map-fusion-based support vector machine(AEKM-Fusion-SVM)method.The experimental results on the SJTU emotion EEG dataset(SEED),BCI2020-A,and BCI2020-B datasets reveal that: 1)the AEKM-fusion method improves the effectiveness and efficiency of the fusion of features of different dimensions;2)the recognition accuracy of the fused features outperforms each individual feature,and this outperformance is more significant in the high-frequency bands(i.e.,the beta and gamma bands);3)the fused features significantly enhance the classification performance for negative emotion;and 4)the fused features built with 27 selected channels achieve comparable performance to that of the fused features built with the full number of channels(i.e.,62channels),allowing for easier establishment of brain–computer interface(BCI)systems in real-world scenarios.Our study enriches the research of emotion-related brain mechanisms and provides new insight into affective computing.Second,we propose a novel comprehensive multi-source learning network(CMSLNet)for multi-modal cross-subject emotion recognition.Specifically,we first design an instance-level adaptive robust consistency metric to align the information between EEG signals and eye movement signals,discovering their consistency and divergences in different emotional aspects.Subsequently,we develop an attentive low-rank tensor fusion method for more effective learning of the intra-modality and inter-modality important features,effectively preventing information loss in the process of fusion.Finally,we utilize domain generalization to extract subject invariant features to adapt to new subjects,increasing the model’s generalization.In these ways,CMSLNet can comprehensively consider the information from multi-source data(i.e.,multiple modalities and subjects),generating more discriminative features for emotion recognition.Experimental results demonstrate that CMSLNet is superior to the state-of-the-art baselines,with high accuracies of 83.15% on the SEED-IV dataset and 87.32% on the SEED-V dataset,respectively,that are 3.74% and4.60 % better than SOTA methods.Notably,the experiments on SEED-IV and SEED-V datasets are shared the same settings of hyperparameters,demonstrating the robustness of CMSLNet.Interestingly,we found that the fused features significantly effectively enhance the classification accuracy of happy and sad emotions.To sum up,this thesis studies the cross-subject emotion recognition of multi-view physiological signals.The proposed AEKM-Fusion-SVM and CMSLNet enrich the research of emotion recognition,effectively increase emotion recognition accuracy.