The Research On Multimodal Fusion Emotion Recognition Based On Deep Learning

With the improvement of hardware and development of the field of machine learning,researchers in different fields are paying more and more attention to the direction of human-computer emotional interaction.Multimodal physiological signals stand out among various types of carriers because they can objectively reflect the true emotions of subjects.Emotion recognition based on physiological signals have attracted more and more scholars' attention in the field of emotion analysis.At present,the physiological signal of the EEG modal is a hot topic in the research of emotion recognition,however the unimodal EEG of the subject has insufficient emotional representation and the physiological signal of the same subject is too much affected by the time period.There are also individual differences in the multimodal physiological signals among subjects.Based on the above three questions,this article has done the following three studies:(1)Aiming at the problem of insufficient emotional representation in the unimodal,a multi-stage multimodal dynamic fusion network(MSMDFN)based on the correlation between modalities is proposed.The network uses the interactive relationship between multimodal features to extract fine-grained emotional information.This method first extracts the primary feature from the original physiological signals of the three modalities(EEG,EOG and EMG).Then calculate the pairwise correlation between the feature,and then perform dynamic multi-stage fusion on the primary emotional features based on the correlation.This method not only extracts the inner-modal emotional representation,but also the inter-modal information and the global information.The multimodal features extracted by this method contain rich emotional information.In order to prove the effectiveness of this method,experiments were carried out on the DEAP dataset.The experimental results show that the physiological signals of the three modalities contains more emotional information than the unimodal and bimodal data.Additionally,the comparison with existing fusion methods have shown excellent emotion recognition ability of MSMDFN.(2)Aiming at the problem of individual differences in multimodal signals between different subjects,a cross-subject multimodal emotion recognition model based on domain adaptation is proposed.The model can be divided into two modules,the multimodal feature extraction module and the domain adaptation module.This method first extracts the preliminary feature of each modal from the original signals through convolutional neural networks,and then fuses the multimodal features through the multi-stage fusion method.After obtaining the multimodal features of the source domain and the target domain through the previous module,the MMD distance is used in the classification layer to measure the distance between the features in different domains.In the training process of the model,the MMD distance and cross-entropy are used as the basis of iteration,so as to realize the alignment of the distribution of the source domain and target domain features,and then improve the generalization ability of the emotion recognition model.In comparison with the non-transfer method and traditional transfer methods,this paper uses MMD-based cross-subject multimodal emotion recognition method to achieve better emotion recognition results on the DEAP dataset.(3)Aiming at the problem of the instability of multimodal physiological signals of the same individual across time periods,this paper proposes a cross-period emotion recognition model that integrates multimodal physiological signals.The model is aimed at EEG and eye movement data,extracts DE features from EEG,and extracts statistical features from eye movement data,and then uses the DCCA fusion method based on the attention mechanism to analyze multimodal features.After fusion,the MMD is used to measure the distribution difference between the source period feature and the target period feature at the classification layer.In the process of model training,based on classification loss,correlation loss and MMD loss,both the multimodal fusion capability and the migration capability of the model are improved.In comparison with the traditional migration method,the cross-period emotion recognition model proposed in this paper shows better emotion recognition ability on two datasets,SEED-IV and SEED-V,and has relatively stable emotions on different subjects.The experimental results of the above methods on the multimodal emotion dataset can show: 1.The multimodal physiological signal contains more sufficient emotion information,and more accurate results can be obtained by using it for emotion recognition.2.Migration technology can make the multimodal emotion recognition model more generalized in cross-period emotion recognition based on specific subjects and cross-subject emotion recognition.