Research On Emotional Recognition Based On Trusted Multimodal Fusion

With the continuous development of information technology,the fields penetrated by artificial intelligence are becoming more and more extensive,and people have higher requirements for human-computer interaction.Users desire that machines can have the ability to observe and recognize human emotions as well,which can greatly improve human-computer interaction and user experience.Researchers use speech,expression,text and other forms of data for emotion recognition research,and multimodal fusion is currently a hot area for emotion research,using the information complementarity between multiple modalities to achieve performance improvement.In this paper,based on the research of unimodal recognition methods for speech and expression images,combined with the research of multimodal strategy fusion,a credible multimodal fusion emotion recognition scheme is proposed,and the main research contents are as follows:Information interactionisinsufficient during traditional speech feature fusion.Therefore,aspeech emotion recognition method based on multi-feature multiplicative fusion is proposed.Firstly,the original speech data are preprocessed to extract the Mel spectrum and linear combination of rhyme features respectively,and the Mel spectrum is passed through SE-Net to get a deeper feature map,then the association between the spectral features and rhyme features is established by decomposing the projection matrix in multiplicative fusion with a low-rank matrix,and finally a neural network is used to realize the emotion classification of the fused features.Experiments on the IEMOCAP and MELD datasets obtained accuracy of 82.5% and78.75%,respectively,which proved the effectiveness of the method.In facial emotion information extraction,the model treats the information in different dimensions and regions equally,while the emotion information is distributed non-uniformly in the feature matrix.Therefore,amulti-scale image emotion recognition method based on CBAM is proposed.The pre-processed map features are convolved by convolutional kernels of different sizes to extract different receptive field information,then the multi-scale feature matrix is obtained through Upsampling and Concat operations,and the convolutional attention mechanism is introduced to learn the sentiment weight distribution vectors under different scales and regions,so that the model can pay attention to the key information,and finally it is connected to the classification output layer for sentiment discrimination.The accuracy reaches 93.5%and 74% on the CK+ and MELD datasets,respectively.The excellent results of the image sentiment classification model lay a good foundation for the bimodal fusion.Based on the single-modal speech emotion recognition method and emotion recognition method proposed in this paper,a multi-feature and multi-decision fusion scheme is proposed by combining the fusion algorithm of decision layer.The outputs of the two modalities are re-modeled,the Softmax activation function is replaced by the Relu activation function to obtain the belief quality of the model,and the uncertainty of the modalities is calculated.The uncertainty is related to the fuzzy propositions in the evidence theory,and the bimodal decision information is credibly fused according to the D-S evidence theory,discarding the conflicting information in the decision,and avoiding the possible paradoxes of the D-S evidence theory in the fusion after the adjustment of the evidence layer and the introduction of uncertainty.Through experiments on the public sentiment dataset MELD,the identification accuracy of the plausible multimodal fusion method proposed in this paper reaches81.25%,which fully demonstrates the superiority of the model in this paper.