Research On Multimodal Emotion Recognition Method Based On Regularized Deep Fusion Of Kernel Machine

The goal of affective computing is to develop artificial intelligence with the ability to perceive,understand,and regulate emotions.One of the most important steps is to achieve the recognition of emotional states.By using the data captured by various sensors to construct a recognition model that can sense and understand the state of human emotions,a more harmonious human-computer interaction environment can be created.Physiological signals reflect the spontaneous movement of the human nervous system,which is difficult to disguise.It has better objectivity than the external features such as expression and voice.Therefore,constructing an emotion recognition model from physiological signal data has attracted wide attention of researchers.However,the construction of the emotion recognition models based on single-modal physiological signals often has the following problems:(1)Physiological signals are significantly affected by individual differences,resulting in unstable model performance;(2)The physiological signal data is susceptible to noise during the acquisition process,which affects the reliability of the emotion recognition models based on single-modal physiological signals;(3)Human emotions are extremely complicated.Relying on single-modal physiological data often fails to model complex emotional states.Constructing an emotion recognition model by combining multiple physiological signals can obtain a better performance model and alleviate the above problems.In the construction process of multi-modal fusion model,the input data often has multi-source and heterogeneous characteristics.However,most of the existing multi-modal fusion models have strict assumptions about the data types of the input data,so that they can only be constructed under certain types of input data and cannot be expanded,which makes the model lose scalability and flexibility.In addition,the correlation and diversity between the fused data makes the simple fusion strategy ignore the complex relationship between different modal data in the fusion process,which limits the performance of the model.Aiming at the problems existing in the process of multimodal fusion,this paper analyses the correlation between physiological signals and emotional states and explores the commonly used multi-modal fusion methods and their advantages and disadvantages.Combining the advantages of fusion methods from different levels,a multi-modal emotion recognition model based on Regularized Deep Fusion of Kernel Machine(RDFKM)is proposed.The main work and contributions are summarized as follows.1.This paper proposes a multimodal emotion recognition model based on RDFKM.For multi-modal data with heterogeneous and multi-source characteristics,RDFKM model uses kernel matrix to construct data representation,expanding the types of input data as model.Using the advantages of deep network model in representation learning and the flexibility of sharing representation fusion method,the model's adaptability to different tasks is enhanced.The above characteristics make the RDFKM model have good flexibility and scalability while improving multi-modal data fusion performance.2.Aiming at the problem that the multimodal fusion model lacks the exploration of the relationship between the fusion data,the RDFKM model explores the correlation and diversity between the fusion representations by performing the norm regularization on the weight of the fusion layer.By controlling the similarity between different weights corresponding to representations,the contribution of the different representations to the fusion representation is changed,and the generation process of the final fusion representation is guided.At the same time,an optimization algorithm based on alternating optimization strategy is proposed,which unifies the process of exploring the relationship between fusion representations and the optimization process of the model.3.The performance of RDFKM model on multi-modal emotion recognition task is verified.Performance experiments were constructed based on EEG,GSR,EMG and RES data from DEAP dataset.Experiments show that the fusion model has better performance than the typical fusion model of other fusion levels.The results of using t-SNE to visualize the fusion representation generated by different fusion models also prove that the resulting fusion representation has higher class separability and better fusion performance.Compared with other existing research results,the proposed method also has performance advantages and reflects the better applicability of the model to different tasks.In summary,the RDFKM model proposed in this paper can deeply explore the relationship between multimodal physiological data and generate an effective fusion representation for emotional recognition task,which has good flexibility and scalability in the face of heterogeneous,multi-source physiological data.It provides a new idea for emotional recognition fusion modeling based on multimodal physiological data.