Research Of Emotion Recognition Based On Multi-modal Fusion

In recent years,Artificial intelligence is applied to our life,,there are more and more scenes of human-computer interaction.If the computer can understand human feelings,it can serve the people more humanized.Emotion recognition is the key technology to achieve this goal.Since multimodal emotion recognition can improve the accuracy of emotion recognition by using the relevance of information between modes,more and more researchers focus on this field.Therefore,it is of great value to study the emotion recognition model based on multi-channel fusion and improve the recognition rate of the model.In this paper,multi-modal fusion of emotion recognition as the research object,in order to improve the accuracy of emotion recognition as the research goal.Based on speech,facial expression of these two modes of research,the main research work is as follows.This paper proposes a decision fusion method based on channel attention and1*1 convolution layer.The channel attention mechanism is implemented by SEblock.The function of SE-block is to give weight to each channel of multi-channel data,and then superimpose and fuse multiple channels to get the emotion recognition results of multi-modal fusion.Through the experimental comparison and analysis of single-mode emotion recognition and multi-mode fusion emotion recognition,it is proved that introducing other modes on the basis of single-mode can greatly improve the accuracy of emotion recognition.At the same time,by comparing the experimental results with the research results of others,it also proves the feasibility and effectiveness of the decision fusion method proposed in this paper.For single-mode emotion recognition,since multiple face images are sampled from the same video,this paper uses merge processing,Max processing and mean processing to further process the features extracted from these face images,so that it can be used for the input of different classifier models.Then,three different classifier models are trained by using speech features and processed facial emotion features.In this paper,we use a variety of features and classifiers combination for experiments,the purpose is to obtain the feature and classifier combination with the highest accuracy of emotion recognition for the subsequent multimodal fusion work.This paper designs and implements a multi-modal fusion emotional recognition system.The decision fusion method proposed in this paper is applied to the computational module of affective recognition in the system.In addition,the system is tested to verify the availability of the system.