Deep Learning-Based Bimodal Emotion Recognition From Facial Expression And Body Posture

Emotional interaction has received great attention in the study of human-computer interaction,and emotion recognition is the key to emotional interaction.Emotion can be expressed in many ways.At present,the research on emotion recognition mainly includes single-modal emotion recognition such as facial expression emotion recognition,speech emotion recognition and body gesture emotion recognition.Psychological research shows that when people make judgments about human communicative behavior,they rely more on the visual channel composed of face and posture than any other channel.Therefore,the study of bimodal emotion recognition based on facial expression and body posture has important practical significance.The application of two modes on facial expression and body posture based on deep learning in emotion recognition is proposed in this thesis.The main work is as follows:(1)Because the traditional methods have the complex process of artificial design features and extracting features,a facial expression emotion recognition method based on deep learning is adopted.Firstly,CaffeNet is improved and a method on facial expression emotion based on improved CaffeNet is proposed.Then,considering that a small number of data samples can also obtain a relatively good recognition effect by using deep convolutional neural network,so a method on facial expression recognition based on VGG-16 network fine-tuning model is studied.Finally,the data studied in this thesis is video sample and the change of facial expression has time correlation,so the facial expression emotion recognition method of long-term recurrent convolutional network is studied.(2)In order to more accurately identify the single-modality of body posture,a body gesture emotion recognition method based on three-dimensional convolutional neural network is proposed.Video clip as a spatio-temporal sequence has strong correlation in time.The three-dimensional convolutional neural network has the advantage of extracting features from video frame sequences,so the video frame sequence intercepted by the video sample of the body pose is input into the different three-dimensional convolutional neural network to complete the classification of the body gesture emotion.(3)Taking facial expression and body posture as research objects,the decision-level fusion algorithm based on weighted summation and the feature-level fusion algorithms of canonical correlation analysis,kernel canonical correlation analysis and multi-set canonical correlation analysis are analyzed.SVM(Support Vector Machine)is used to classify the features of the fusion of the two modes.In this thesis,FABO(A Bimodal Face And Body Gesture Database)is selected for experimental verification.The experimental results show that the accuracy of bimodal emotion recognition obtained by appropriate fusion method is about 4%~5% higher than that of single-modal emotion recognition.