Emotion Analysis And Recognition Based On Eye-tracking Data During Free-viewing Of Omnidirectional Images

With the development of 5G and VR,applications based on omnidirectional(also called360-degree)content have received widespread attention.Studying the user's perception and emotional response to omnidirectional content is essential for multimedia processing and user quality of experience assessment.The eye movement characteristics will change significantly under different emotional omnidirectional content stimuli,and eye movement monitoring has the advantages of convenience and authenticity so it has attracted much attention in the field of emotion recognition research.The current emotion recognition research based on omnidirectional content-induced eye movement monitoring has faced with the following problems:(1)There are few eye movement datasets with emotion tags published under omnidirectional content,and related studies often use head movement to replace eye movement data;(2)Research shows that there is a relationship between eye movements and emotions in 2D content,but 3D panorama-induced state eye movement mode is not the same as 2D static content.How to extract eye movement features and analyze the relationship between eye movement bais and emotions in omnidirectional image in a free-viewing way;(3)Previous studies were mostly based on raw eye-tracking data,lacking the extraction of eye movement behavior features,and failing to fully mine temporal correlation information of eye movement sequences.In response to the above problems,the following work was carried out in this paper:(1)Based on the omnidirectional images created by the LS2 N laboratory saliency360 dataset,the stimulus materials were selected containing emotional tags(positive,neutral,negative)and emotional eye-tracking 360-degree dataset was built,including head movement and eye movement data of 19 subjects based on HTC Vive headset and SMI Eye-tracker device.(2)According to the interactive characteristics of virtual reality headsets,this thesis proposes an eye-tracking data processing framework in free-view mode of omnidirectional pictures,using viewport mapping and method based on time and speed thresholds to extract eye movement behavior features.ANOVA analysis and Dunn multiple comparison analysis show that in 3D free-viewing way,negative images get fewer gaze points than neutral pictures,which is different from the observation results of static pictures.It is inferred that the discomfort caused by omnidirectional images will bring more avoidance eye behavior,specifically showing a longer,larger and faster saccade.(3)This thesis has conducted sufficient experiments on the emotion classification of different algorithm models and different eye movement features,and verified the effectiveness of eye movement behavior features.Among them,the SBFS-GBDT method can obtain an accuracy rate of 79.12% in the positive and negative two classification experiments,which is better than the traditional method.(4)Using the interdependence characteristics of the eye movement sequence,the GRU algorithm based on the eye movement scanpath is constructed.The segmentation sequence method is used to effectively use the context information.In the positive and negative binary classification,it can be improved by 2.4% compared with the previous algorithm,which verifies the algorithm's superior performance.This thesis has carried out research on emotion analysis and recognition based on eye movements under omnidirectional images.The dataset created can promote the development of this field.The proposed algorithm provides a reference for future research and promotes the development of emotion recognition based on eye movement in omnidirectional content.