A Cross-culture Study On Emotion Recognition Using EEG And Eye Movement Signals

Emotion is a mental state related to thinking,perception and behavior.Exploring the mechanism of emotions and the characteristics of emotions under various conditions can help people better understand themselves,provide technologies for the development of more friendly human-computer interaction systems and help achieve objective assessments and treatments of mental illness.Cross-cultural research on emotional differences is also an important branch of research in psychology,cognitive science,neuroscience,and psychiatry.Many scientific studies have found that for people of different cultural backgrounds,there are similarities and differences in understanding emotions.However,most of the existing researches are based on qualitative analysis,but lack of objective and quantitative research.Therefore,this study selected Chinese and French to conduct a comparative study of positive,neutral,and negative emotion recognition task by using EEG and eye movement signals.This study aims to explore the similarities and differences between the EEG and eye movement signals' patterns of Chinese and French subjects.The data of Chinese subjects are selected from public dataset.Therefore,we designed the experiment and collected stimulating materials for French subjects.We invited 8 French subjects and completed a total of 24 experiments.The EEG and eye movement data collected in the experiment will be pre-processed,feature extracted and feature smoothed,and input to the classifier for classification.In order to explore the similarities and differences between Chinese and French subjects,we used support vector machines to classify EEG and eye movement signals and used brain electrical activity mapping and functional brain connectivity analysis to visualize.In order to explore multi-modal cross-cultural emotion recognition and improve the accuracy of cross-cultural emotion recognition,we use deep canonical correlation analysis to fuse EEG and eye movement signals.To solve the problem of the lack of training data,we used conditional Wasserstein generative adversarial network to generate EEG and eye movement data.Through comparison of experimental results,we found that the EEG features of Chinese and French subjects have better ability to distinguish the three types of emotions in the high frequency band.Among different EEG features,the differential entropy features are the feature that can obtain the highest accuracy.The results of visualization show the similarities between Chinese and French subjects in the higher frequency bands.Using conditional Wasserstein generative adversarial network effectively generated EEG and eye movement data.We found that EEG features have an advantage in distinguishing positive and neutral emotions,while eye movement features are better at distinguishing negative emotions.After the fusion of two modalities,the accuracy of emotion recognition of both Chinese and French subjects has been improved.