Research On Multi-modal Fusion Emotion Recognition

Emotion plays an important role in human communications. Changes of affectivestates have an impact on people’s perception and decision. Emotion recognition is asignificant research field of pattern recognition. It introduces emotion intohuman-computer interaction. People express their emotions through facial expressions,speech, postures, physiological signals, characters, and so on. As a result, emotionrecognition is inherently a matter of multi-modal fusion.In this paper, we construct a framework of multi-modal fusion emotionrecognition. Facial expression features and speech features are respectively extractedfrom image sequences and speech signals. An emotion classifier is designed to fusefacial expression and speech modalities based on Hidden Markov Models andMulti-layer Perceptron. In order to locate and track facial feature points, we constructan Active Appearance Model for facial images with all kinds of expressions. FacialAnimation Parameters are calculated from motions of facial feature points asexpression features. We extract short-term mean energy, fundamental frequency andformant frequencies from each speech frame as speech features. We use expressionfeatures and speech features to train Hidden Markov Models based on ViterbiAlgorithm. Multi-layer Perceptron which fuses expression and speech modalities istrained based Back-propagation Algorithm. Experiments indicate that multi-modalfusion emotion recognition algorithm which is presented in this paper has relativelyhigh recognition accuracies.The proposed algorithm makes use of affective information from video and audio.The approach has better performance and robustness than methods using only videoor audio separately.