Dual-mode Video Expression Recognition Based On Facial Expression And Physiological Signals

With the continuous development of artificial intelligence,people hope that computers can understand the emotions conveyed by humans in the process of interaction and can naturally communicate with human beings with emotions.Therefore,emotion recognition has gradually become one of the key points of research to achieve harmonious human-computer interaction.To fully exploit the emotional information of face video,video emotion recognition research is carried out from two aspects of expression of video facial and physiological signal of facial blood volume pulse in this paper.The main work and innovation of this article are described as follows:(1)Against the problem of fewer international public database of video emotion currently,a complete set of emotional evoked experiments was designed.The experiments collected 30 subject's emotional videos.And then some preprocessing operations were performed for videos such as clip and so on to obtain the database that can be used for emotion recognition.The database collected a total of four emotions,incd uding happiness,fear,sadness and anger.(2)The previous methods of video emotion recognition mainly transform the color image in the video into a grayscale image,and then extract the facial features from the grayscale facial image,and pay little attention to the emotional factors involved in the color information in the video.In addition,common physiological signals are obtained through specialized physiological collection instruments in a controlled environment.All of them are contact-type,and the data collection process is relatively complicated,and it is easy to bring discomfort to the subjects.To fully exploit the emotional information involved in the face of videos,a recognition method for video emotion based on physiological signals of blood pressure pulse(BVP)of the human face is proposed,the physiological signal acquired without contact.First of all,the method enlarges the color on the face image of the video to amplify the weak BVP signal,and then extracts the time domain,frequency domain,and nonlinear characteristics of the video.Considering the influence of facial expressions,glasses,and bangs on the quality of physiological signals,this method divides the face region and tests and analyzes different regions and different emotional characteristics.The experimental results prove the feasibility and effectiveness of the proposed method.The experimental results prove the feasibility and effectiveness of the proposed method.(3)Aiming at the limitations of single-modality in emotional expression,a dual-mode video emotion recognition method for merging facial expression and BVP physiological signal is proposed.First,the method extracted the LBP-TOP and HOG-TOP features and BVP physiological signals of human face in the video respectively.And then integrated LBP-TOP and HOG-TOP features for expression recognition and used facial BVP physiological signals for emotion recognition.Finally,the two sentiment recognition results are sent to the fuzzy integral fusion classifier for dual-modality fusion emotion recognition,the fusion result obtained.The experimental results show the effectiveness of the bimodal fusion method.Comparison with other literatures shows that this method has certain advantages.