Research On Emotion Recognition Methods Based On Facial Expressions And Physiological Signals

As we all know,emotion is so important that it influences our lives in many different ways.With the development of computer science and electronic information,emotion recognition based on computer now becomes a significant research for researchers.Facial expression is the most direct way to express one's emotion,so it was used to emotion recognition at first.However,facial expression is an element that can be pretended,which means it's not objective and can not be totally trusted.On the other hand,physiological signal is an objective element because no one can pretend their heartbeat,therefore,emotion recognition based on physiological signal are becoming more and more popular.But most of the physiological signal is collected by wearable devices or electrodes that touch one's body directly,which can affect the subject's emotion.So a non-contact emotion recognition system based on both facial expression and physiological signal was proposed,in which the facial expression signal was obtained from camera and the physiological signal was obtained from continuous wave radar.High accuracy was achieved by comparing the classification on each signal and feature fusion.The main works of this paper is as following:1 The method to extract face from video was introduced,the traditional method on emotion recognition through facial expression,LBP feature with SVM,was researched.And the emotion recognition from facial pictures based on deep learning stacking model was proposed.The results showed that the accuracy by the method which was proposed was 12.9%higher than the traditional method.2 The theory of non-contact physiological signal obtained by continuous wave radar was introduced.Traditional features extracting from physiological signal was researched,and the time-varying features based on Complete Ensemble Empirical Mode Decomposition with Adaptive Noise,CEEMDAN,and Hilbert Transform extracted from physiological signals were proposed.3 A non-contact emotion recognition system based on both facial expression and physiological signal was proposed.For physiological signals,feature extraction was carried out first,and then the method of feature selection based on random forest was adopted to select the physiological signal features and then emotion classification was carried out.As for facial expression signals,the features were extracted first and classification was carried out.Then,the classification results of physiological signals were compared with those of facial expression signals.If the results were same,it was the final emotion recognized by the system,otherwise,the features of the two sensors were fused and the result were calculated by the random forest.4 Emotion recognition experiments were designed and conducted.Collect non-contact physiological signals and facial expression signals of subjects simultaneously in happy,sad,fear and neutral states.After processing the two kinds of signals separately,the accuracy rate of facial expression signals was 42.2%.And the accuracy of the physiological signals was67.7%.When time-varying features were added into the physiological signals,the accuracy was improved to 70.5%.After the feature selection of the physiological signals,the accuracy was improved to 72.1%.Finally,the results of emotion classification of the two signals were compared.When the results of emotion classification of the two sensors were the same,which in these experiments were 35.8%,the accuracy rate was 96.0%.When the results of emotion classification are not the same,which in these experiments were 62.7%,the accuracy of emotion classification was 74.2%,which was processing through random forest.And we could calculate that the final accuracy is 82.3%.