Research On Multimodal Emotion Recognition Based On Feature Fusion

Emotion recognition is an important branch in the field of affective computing,and it is a hotspot in the field of signal processing,pattern recognition,artificial intelligence,human-computer interaction and so on.Different modes of emotion recognition complement each other,and constantly improve the accuracy of emotion recognition system,prompting the emergence of high-quality,more harmonious human-computer interaction system,so that human life becomes more intelligent and convenient.In this paper,we mainly study the emotional recognition system based on facial emotion characteristics,phonological emotion characteristics,postural emotion characteristics and physiological signal-related emotion characteristics,including emotion feature extraction,multimodal emotion feature fusion and emotion feature classification and recognition.The main work of this paper is as follows:(1)Extracting the characteristics of different modes of the data in the database,including: extracting the Gabor feature of the facial expression data using the two-dimensional Gabor filter;extracting the integrated speech emotion characteristics(including amplitude,fundamental frequency,formant,etc.)using OPENSMILE;extracting its temporal and spatial characteristics by finding the temporal and spatial points of the emotional data.(2)The existing patterns are used to fuse the features of the different modes extracted,including: principal component analysis(PCA),kernel correlation analysis(KCCA)and kernel matrix fusion(KMF).On this basis,a new method is proposed,genetic algorithm are applied to fusing the features.The experimental results show that the proposed method has better fusion effect.(3)The characteristics of single modal are identified by support vector machine(SVM)to realize single-mode emotion recognition,and the results of single-mode emotion recognition are obtained.On this basis,the fusion feature obtained by different methods are identified by support vector machine(SVM)to realize multi-modal emotion recognition,and the results of multi-modal emotion recognition are obtained.The experimental results show that,from the point of view of the recognition results,the results of multimodal emotion recognition using the appropriate fusion method are better than those of the single modal emotion recognition.