Research And Application On Emotion Recognition Algorithm Based On Multi-modal Eye Movement Information

Emotion recognition refers to inferring the emotional state of a person based on observing,analyzing,and recognizing the effective information of the emotional state.With the rapid development of computer and information technology,emotion recognition has been widely applied in many research fields,for example,medical diagnosis,video surveillance,intelligent education,and human interaction,etc.Compared with emotion recognition based on biological features such as expressions,voices,gestures,etc.,emotion recognition based on physiological features can reflect the most realistic emotional state of a person more objectively and accurately.As a matter of fact,the eye movement information of the physiological signal can reflect the state of mental activity of the person to a certain extent.By acquiring eye movement signals and analyzing their potential eye movement information,it can provide more clues for emotion recognition.Therefore,emotion recognition based on eye movement has important significance and value for research and application.This thesis focused on using the combination of EOG(Electrooculogram,EOG)signals and VOG(Video-oculography,VOG)signals for preprocessing,feature extraction,feature fusion,classification and recognition,as well as application in three emotional states.The specific research works are as follows:(1)Research on the existing emotion recognition technology and emotion recognition methods based on images and physiological features.Considering the advantages and disadvantages of EOG signals and VOG signals,this thesis proposed the combination of EOG and VOG to collect eye movement information.The collection method had the advantages of simple operation,low experimental cost,and small environmental impact.In order to effectively induce emotional state and improve the quality of acquired signals,this thesis designed a stimulus source selection method and a bimodal acquisition paradigm,and six subjects were collected,each one had a total of 72 one-minute eye movement data for each emotional state.(2)Research on the ability of eye movement features for emotions expression.For the EOG signals,we extracted the time/time-frequency/spatial features,respectively.The saccades time in the signal segment was detected by continuous wavelet transform,on this basis,we extracted the maximum,mean,standard deviation,and frequency of saccades time features of the saccades time;and the time-frequency domain feature of the EOG signal was extracted by short-time Fourier transform;Similarly,the spatial domain filter design was performed using independent component analysis,and the spatial domain feature was extracted on this basis.For the VOG signals,the Hough transform was utilized to calculate the diameter of the pupil in the segment and the maximum,mean and standard deviation features of the pupil diameter were extracted.In addition,the fixation signal segment was detected by calculating the change of the pupil center point in the continuous frames,and the maximum,mean values standard deviation frequency of fixation time features were extracted.This thesis also studied the impact of different emotional time lengths on emotion recognition.The above features were extracted for five different emotional data lengths i.e.1 second,2 seconds,4 seconds,6 seconds,and 10 seconds.By comparing the recognition results under different time lengths,when the time length of the emotion sample was 2 seconds,the average recognition rates of all features were relatively good.The recognition results corresponding to different eye movement features were as follows:the correct rate based on the saccade feature was 54.6%,fixation was 55.94%,pupil diameter was 56.18%,time-frequency domain was 47.1%,the spatial domain was 52.5%,respectively.(3)Research on two feature fusion methods of feature level and decision level.The method of feature level fusion included the fusion of feature direct fusion and PCA(Principal component analysis,PCA)for dimension reduction.Decision level fusion was a fusion of certain posterior probabilities obtained by classifying each feature,the fusion rules included maximum,minimum,mean,sum,and product rules.The fused features were classified using a support vector machine,where the average recognition result of PCA method was 57.89%,which was 0.49%higher than that of direct fusion,and the average recognition result based on mean and summation in decision level fusion was 67.86%.which was 9.97%higher than the feature of direct fusion multimodality.The experimental results showed that the multi-modal feature fusion method based on decision level fusion was better than the direct feature fusion method.(4)Design and implementation of a video content evaluation system based on eye movement information.For the evaluation of video content,it is mainly relying on manual implementation.In order to improve the evaluation efficiency,this thesis designed and implemented a video content evaluation system based on eye movement information.The system was mainly composed of functional modules such as signal input,preprocessing,feature extraction,model training and video testing.The test recognition results of each subj ect were higher than 70%,which showed that the system was stable,with strong interaction and simple operation.