Dynamic Video Brain Information Classification Decoding Based On Dilated Long And Short Term Memory Neural Network

Decoding visual perception content from Functional Magnetic Resonance Imaging(fMRI)is an important research topic in the field of brain-computer interaction.However,due to the small sample size,large noise,high dimensionality and high acquisition cost of fMRI data,the visual decoding performance based on brain signals is low.The underlying visual neural coding and decoding mechanism needs to be further explored.At present,there are many studies on visual perception decoding of still images,but less research on visual perception decoding of dynamic video.This is due to the temporal variability and content complexity of dynamic videos.Dynamic perception and information integration are the basic forms of human understanding of the world,and studying the neural encoding and decoding mechanism of dynamic visual information allows us to better understand how the brain works.In this thesis,dynamic video is used as visual stimulus.FMRI technology is used to classify and decode the brain information of dynamic video based on the dilated long and short term memory network,and the main contents include:(1)Three fMRI experiments were conducted,namely the retinal topology experiment,advanced visual brain region localization experiment,and dynamic video decoding experiment.In the retinal topology experiment,a checkerboard stripe stimulation was presented to participants to establish the mapping relationship between visual stimulation and fMRI signals in the lower visual cortex,allowing for the delineation of the low-level visual areas.In the advanced visual brain region localization experiment,the mapping relationship between complex visual stimuli and the fMRI signal response of the higher visual cortex was established by viewing a series of natural pictures for the subjects,and the advanced visual brain area was divided.In the dynamic video decoding experiment,participants watched a 2-hour dynamic video consisting of five categories and acquired their fMRI signals.(2)According to the visual information integration mechanism,a dynamic video brain information classification and decoding model based on Dilated Convolution Long and Short Term Memory(DC-LSTM)is proposed,and multi-scale time information is extracted from brain signals and fused by dilated convolution with different coefficients.By comparing the decoding accuracy corresponding to different dilated coefficients,it is found that for short and medium time series brain signals,the best performance is performed when the time integration scale is 4.At the same time,by comparing the classification and decoding performance of single-time point brain signal,average brain signal and scrambled time series brain signal,it is found that temporal information integration can extract brain information more related to the stimulus category,so as to obtain better decoding performance.In this thesis,the decoding accuracy of low-level and high-level visual cortex is further compared,and it is found that the high-level visual cortex has stronger information integration ability.Finally,this thesis analyzes the characterization characteristics of different categories of videos by using the representation dissimilarity matrix,and provides evidence for the consistency of brain activity patterns and visual stimulus feature drive.In summary,this thesis proposes a brain information classification and decoding model based on dilated long and short term memory network,which better decodes the brain signals of subjects when watching different types of dynamic videos,and provides technical support for the development of dynamic visual perception decoding.