Realization Of Multi-modal Emotion Recognition Based On Speech,Expression And Gesture

Speech,posture,and facial expressions are considered to be the main channels of human emotions in human social activities and daily communication.In recent years,the research of human emotion recognition has made great progress,which laid a solid theoretical foundation for the future popularization of artificial intelligence.At first,research on human emotional recognition still stay in the single-mode emotional recognition,however,in recent years,with the continuous development of science and technology,especially artificial intelligence technology,more and more researchers put their eyes on multi-modal fusion.This paper presents a three-mode emotion automatic recognition method based on speech,expression and posture.The main contents of this paper are as follows:(1)Firstly,the acoustic feature of the speech signal and the MFCC feature are extracted;for the expression signal,the Gabor feature is extracted by Gabor wavelet transform,and the feature is reduced by the principal component analysis method to remove the redundancy information,which make it easier to handle;for the posture signal,using the EyesWeb platform to extract the feature of the posture.(2)The emotional features of the modalities were normalized,and the emotional features of three modalities were combined by Discriminative Multiple Canonical Correlation Analysis(DMCCA).Then,the feature selection algorithm Relief F was used to fuse Post-feature features.(3)After obtaining the emotional features,the training model based on the support vector machine is trained to obtain a prediction model,and the prediction model is used to judge the emotional category label.