Research Of Facial Expression Recognition Based On Image Sequence And Audio Emotion

With the improvement of human nonverbal communication and emotional ability,emotion recognition has made great progress in the field of human-computer interaction,virtual reality,augmented reality,and computer-aided education and so on.Emotional intelligence has gradually become a focus of research in the field of artificial intelligence.The content of emotion recognition research includes facial emotion,speech emotions,physical behaviors,text classification,physiological signal recognition and many other multi-modality human behaviors.Through these contents,the user's emotional status can be identified.At present,the researches on facial expression recognition are mainly focusing on facial expression recognition at domestic and foreign,but less on speech emotion recognition.Extracting the feature of dynamic expression change of facial expression in image sequences can improve the recognition rate.The robustness of expression recognition can be improved by combining the result of audio emotional classification with expression recognition of image sequence.The main content of this paper includes:Firstly,in order to extracting features related to facial expression action units,a multi-scale deformable component model(DPM)is proposed to detect and locate key parts of face,the detection accuracy rate can reach 97.6%.We can further extract facial expression features of the same dimension for local regions of multi-scale images.The redundancy of the facial expression features can be effectively reduced by detecting the local area and then extracting the features from the area.Secondly,traditional facial expression recognition based on static images,such as Gabor filter sets to extract global features of expressions and histogram of gradient direction(HOG)to extract local features,both of them are not very effective in facial expression recognition.In this paper,two methods are proposed for the study of image sequence expression recognition: The first is to use the optical flow method to learn the dynamic features with the LK feature point tracking method,which ensures the information transfer between the image frames and frames.Then the classifier is trained using AdaBoost algorithm.The second is to use timedependent short and long-term memory neural networks(LSTM)based on Deep Learning TensorFlow Architecture to extract dynamically changing facial expressions in the image sequence to train LSTM Model.Compared with the static image,the two methods of image sequence have a great improvement in facial expression recognition.Finally,in the research of audio emotional classification,we used the Mel-scale Frequency Cepstral Coefficients(MFCC)to extract audio features,and the model was trained using AdaBoost classification algorithm.Finally,the result is using different weighted combine facial expression recognition of image sequence result with audio emotional classification,which effectively improved the accuracy and robustness of facial expression recognition tasks.