Emotional Speech Recognition Based On Facial Expression Analysis

With the wide applications of computer technology in various fields, Speech Recognition becomes more and more important to people recent years as the key technology of Man-Machine Interaction. In order to fully reflect the speaker's message, the study of AVSR (Audio Visual Speech Recognition) has become a hotspot. ESR (Emotion Speech Recognition) is a branch of AVSR, it recognizes the content of voice and speaker's emotion at the same time. This paper constructs AVSR system with an ISR (Isolated Speech Recognition) system based on VQ (vector quantization) and FER (Facial Expression Recognition) based on PCA (Principal Component Analysis).A simple semantic model is put forward in this paper, the main constitutions of this model are word matrix and sentence matrix, they correct deviation of acoustic model by minimum-distance criterion and maximum probability criterion in template matching. Experiment result shows that recognition rate is improved with the help of this simple semantic model.We get facial expression pictures from continuous video streams which recorded by camera. After preprocessing and Principal Component Analying, we determine which emotion kind the picture belongs to by matching up with the template library.In order to realize synchronization between ISR results and FER results, we use punctuations which produced in ISR system to control the times when FER system gets picture from video streams, and fuse those information by marking punctuation and picture. Experiment results show that this integrated system can reflect speaker's emotion while recognizing speech.