Research On Bimodal Emotion Recognition Based On Facial Expression And Speech Signal

The traditional process of human-computer interaction is mainly conducted through the mouse, keyboardand,etc., which is only based on a logical way. In this way computers cannot understand and adapt to human emotions. It can help the computers to understand the emotions and make a corresponding feedback like people to add emotion recognition to computers. This can make the human-computer interaction more friendly and natural and achieve a better user experience. Thus, the study on emotion recognition has attracted many researchers these years. Emotion is an important means of human communication, it can help the communication more smooth and efficient when you obtain other’s emotional information and analyze it. During the conversation, people’s speech, facial expression and postures are all a part of the information, and all of them contain emotional information. Researchers have conducted extensive research on single-mode emotion recognition such as facial expression recognition, speech emotion recognition, etc. However, the single mode using relatively simple emotion features has some limitations on recognition. A research on bimodal emotion recognition combining facial expression and speech is presented in this paper on the basis of existing theory and research. The work of this paper includes the following 3 aspects.(1) Facial expreeeion recognition based on imagesIn this paper, both color images and depth images are used to extract emotion features of the facial expression and facial pose is evaluated roughly according to the facial points. After the preprocessing of the images, Gabor wavelet is adopted to extract features from the color images and statistical distribution of depth value is extracted from the depth images. These two kinds of features are connected as the emotion features and PCA method is used to reduce the dimension of features. Then, SVM is adopted for training and classification for facial expression. The experiment results indicate that the proposed method in this paper has obtained good results.(2) Speech emotion recognition based on Hidden Markov ModelVoice signal is processed in this paper. Emotion features is extracted from multi acoustic parameters, including speed, frequency, energy and MFCC of the voice signal. Then, HMM is used for training and classification for emotion recognition. Through the expriments, the results demonstrate that the proposed method in this paper has achived a good performance in the emotion recognition.(3) Bimodal emotion recognition combining facial expression and speechThe method of decision level fusion is adopted in this paper for bimodal emotion recognition combining facial expression and speech. Bimodal emotion recognition experiments were performed according to the sum rule and product rule respectively. The results show that the proposed method improves the recognition performance significantly.