Cognitive Computing And Collaborative Interaction Research On Fusion Of Eye Movement And EEG Data

Eye tracking can record the user's eye movement data and characterize the user's visual cognitive activity.It can be used to analyze the user's visual attention process and predict the user's interaction intention.The EEG data represent the bioelectrical phenomena inside the human brain.Feature extraction of EEG data can effectively analyze the cognitive process of users and it can be used to implement brain-computer interaction.Therefore,this paper proposed a cognitive computing method based on eye movement and EEG data fusion.By separately collecting the eye movement and EEG data of the user during the motor imagery,the corresponding preprocessing and feature extraction were performed,and then the cognitive calculation such as interaction intention inference was performed based on the data fusion method.Finally,a collaborative interaction protoype system was build and used to complete the interaction task and improve the quality of collaboration.The research work of this paper mainly includes the following three parts:(1)Feature extraction and application of eye movement and EEG data.The motor imagery experiment was designed,and recorded eye movements and EEG data for preprocessing,feature extraction and classification.According to the image processing method,the eye movement features of the user were obtained.The classification accuracy based on pupil diameter,fixation point coordinates,saccade,gaze time and gaze frequency were compared.The results showed that pupil diameter,fixation point coordinates and saccade were closely related to the user's motor imagery,and the average classification accuracy could reached 75.54%,72.23% and 69.11%,respectively.A variety of classification models were constructed for EEG data.The experimental results showed that the classification accuracy of feature extraction was higher than that without feature extraction,and the classification accuracy based on CNN was about 6% higher than that of SVM.The average classification accuracy of the WT+CNN model was up to 80.09%.(2)A fusion method based on eye movement and EEG data.In order to improve the classification accuracy of motor imagery,a data fusion method based on eye movement and EEG was proposed.The eye movement and EEG data were separately combined into the feature level and the decision level.The experimental results showed that the average classification accuracy of the feature level reached 81.16%.The average classification accuracy of the decision level reached 82.56%,which was higher than the classification accuracy of individual eye movements or EEG.At the same time,analyzed the situation of absent EEG channel,and proposed the compensation method of eye movement for EEG.It was validated that the EEG classification accuracy of eye movement plus 2 channels was similar to the EEG classification accuracy of 4 channels.(3)Designed and developed a human-computer interaction system based on multi-users collaboration.This paper constructed the offline training model of the user for the motor imagery,and found out that the classification accuracy of the multi-users data fusion system was higher than that of the single user.At the same time,the offline model was applied to the real-time interactive system,and the Tetris game based on multi-users collaboration was designed.The two users controlled the game object through motor imagery.User experiments' results showed that the system effectively improved the efficiency of collaborative interaction and improved the user experience.