Research On Emotion Recognition Based On Deep Learning And Eye Gaze Signals

Emotion is a kind of complex psychological subjective experience when people experience internal and external stimulation.It is often accompanied by multi-level physiological response and plays an important role in interpersonal communication.With the rapid development of computer and human-computer interaction(HCI)technology,it has become a core issue that how to accurately identify human emotions to build a more friendly and natural HCI environment.As external physiological signal reflecting human cognitive process,eye gaze is of great significance for building a more harmonious HCI environment,forming a more intelligent system for disease diagnosis and treatment and automatic driving.In the field of emotion recognition based on eye movement signals,existing models suffer from poor generalization performance and low accuracy.This is caused by the problems of ignoring the continuity and the high-level features of eye gaze,and rarely using the superlearning ability of neural networks.In this paper,a novel method of feature extraction and emotion recognition based on eye gaze is proposed to solve these problems.In addition,based on the theory that emotion can be expressed by multiple modes,a network-level feature fusion algorithm based on eye gaze and EEG signal is proposed to solve the problem that the existing algorithm fails to highlight the emotional features with strong representation ability and ignores the complementarity of features in the original dimension.The contributions of this paper are summarized as follows:(1)This paper presents a novel method of emotion recognition based on eye gaze.Firstly,in order to reflect the temporal continuity of eye gaze,a novel feature set is proposed,which includes gaze sequence,statistical feature sequence and spectral feature sequence.In addition,an innovative emotion recognition method based on shallow convolution neural network with identity mapping is proposed.This network can extract the high-level features of eye gaze and concatenate them with the original features through identity mapping to obtain a feature set with more emotion representation ability.Compared with the most advanced algorithm,in the MAHNOB-HCI data set,the accuracy of emotion recognition increases by 11.7% and 10.5% respectively in valence and arousal,which proves the effectiveness of the proposed method.(2)Using the complementarity of different modes in emotion representation and combining with the useful information related to emotion in eye gaze and EEG,a network-level feature fusion method based on eye gaze and EEG is proposed.In the proposed method,multi spatial self-attention is used to extract the higher-level features of eye gaze and EEG which can highlight the emotional state.The joint-attention makes the features of the two modes have better complementarity in high dimensions,thereby enhancing the ability to express emotions and further improving the accuracy of emotion recognition.Aiming at the problem that the traditional experimental setup is inappropriate,a more reasonable experimental setup is adopted to improve the generalization performance of the algorithm.Finally,the accuracy of emotion recognition in SEEDIV dataset reaches 73.03%,and in MAHNOB-HCI dataset,the accuracy is increased to 82.32% and 77.96% respectively in valence and arousal,which verifies the effectiveness of the proposed method.