Multimodal Emotion Recognition Based On Face And Speech And The Application In Reasoning Of Robot Service Tasks

As service robots play an important role in family situations,natural human-computer interaction has become one of the key factors affecting user satisfaction and the comfort of human-computer coexistence.How to pay attention to the recognition of user emotions in the process of human-computer interaction,and on the basis of understanding the user's emotional state,let the robot provide a comfortable and good service matching the emotion in the complex and changeable home environment,which has been praised by research scholars.extensive attention.Since the expression of human emotion is a complex and continuous process,the current focus on human emotion recognition is gradually transitioning from discrete emotion recognition to continuous emotion recognition.At the same time,single-modal continuous emotion recognition has the disadvantages of low recognition accuracy and poor robustness.To further improve the accuracy of emotion recognition and enhance the robustness of the recognition system,it is necessary to explore the complementarity between various modalities to improve the final quality of emotion recognition.This paper mainly studies multi-modal emotion recognition based on the fusion of facial expressions and voice modalities at the decision-making level,and explores and experiments on the uncertain service reasoning of service robots in complex home scenarios.The main contents are as follows.(1)In order to solve the problem of scarcity of continuous facial expression emotion data sets and abnormal video frames,this paper proposes a facial expression recognition algorithm based on Gabor transform.Firstly,the original expression video is divided into frames,and then the designed convolutional neural network is used to detect the face of the video frame and normalize the data.Secondly,the Gabor transform is used to extract the emotional features of the processed video frame,and finally with the help of deep learning algorithms realize emotion recognition of facial expressions.(2)In order to improve the accuracy of speech continuous emotion recognition,this paper studies the speech emotion recognition method based on transfer learning to realize the recognition of the user's emotional state.Firstly,the one-dimensional speech signal is converted into a two-dimensional signal by using the Mel frequency cepstrum coefficient,and the emotional feature extraction is completed.Then the extracted features are appropriately cropped.Finally,referring to the residual network,a transfer learning network suitable for continuous speech emotion recognition is designed to realize continuous speech emotion recognition.(3)Research on multi-modal fusion emotion recognition.Due to the limited improvement in the accuracy of single-modal continuous emotion recognition,this paper takes into account the complementarity between the various modalities,takes the two signals of expression and speech as the research object of multi-modal fusion,and analyzes and compares multiple linear regression and Karl Mann filtering two decision-level fusion algorithms.Finally,the multi-modal fusion emotion recognition algorithm is verified on the open database.The experimental results show that the multi-modal fusion method proposed in this paper is better than the single-modal emotion recognition accuracy.(4)Research on the uncertainty reasoning method of robot service task based on emotional feedback.The reasoning result of the service robot is affected by various uncertain factors in the family situation.By analyzing the influence of various factors in the family context on the results of service reasoning,the multi-entity bayesian network is used for robot task reasoning,and the characteristics of its probability graph are used to construct a family context model containing uncertain information.In addition,in order to make the provided service more humane,the user's emotional state is added to the model,and combined with the connection tree reasoning algorithm to realize the uncertainty service task reasoning of the service robot in the complex and changeable home environment.