Bimodal Emotion Recognition Based On Facial Expression And Speech

Emotion recognition is the emphasis of the research of emotional computing over a decade.The rapid development of relative technology will help to realize the perfect harmony of human-computer interaction experience,and bring the people who live in the information age.There are various ways for people to express their feelings,and facial expression and speech are the two which are the most direct and obvious.This paper is to do the bimodal emotion recognition based on facial expression and speech and to explore the method of feature extraction and modal fusion.The specific work is as follows:1.In terms of the bimodal feature extrantion,Gabor and block LBP algorithm are used to extract the facial expression feature.Then,to slove the problem of huge dimensions of Gabor feature,downsampling and extracting the Gabor feature of feature points are proposed in this paper.For extracting speech feature,open SMILE toolbox is used to choose the proper feature set and extract the relative features.2.The method of bimodal fusion is the emphasis of the bimodal emotion recognition.First,a kind of feature-level fusion method based on kernel matrix is proposed,which creates the new feature by doing the weighted-sum to the kernel mapped feature.Then it proposes the fusion method based on multiple kernel learning to improve the KMF method,which allocates series of different kernels to different modals.Use the linear combination of those kernels instead of the kernel in SVM.Finally,the weighted-sum and product prinple is used to do the decision-level fusion based on posterior probability.3.A lot of experiments on e NTERFACE'05 database are excuted to test the performance of recognition.The results of experiments demonstrate that Gabor feature is much better than LBP feature when used in facial expression emtion recognition.What's more,the feature-level fusion method based on multiple kernel learning is the best modal fusion method,which make the resule reach up to 84.23%.This result is 11% higher than the result of single modal emotion recognition.