Research On Deep Network Model Of Visual Cortical Information Representation Based On FMRI

The visual system of human brain is a complex intelligent system that has the efficiency and robustness unmatched by existing machines when processing external visual information.At present,functional magnetic resonance imaging?fMRI?is the primary tool for observing brain activity.It can perform three-dimensional and non-destructive imaging of human brain with high spatial and temporal resolution.The fMRI-based analysis of visual information aims to construct computational models from the characteristics of visual cortical information,which explores the relationship between visual stimuli and corresponding brain responses.Recent research results show that based on certain visual information processing mechanism,the deep neural network models can achieve rich and hierarchical feature expressions,which has obvious advantages in fMRI-based visual information analysis.Studying the deep network models of visual information representation based on fMRI is of great significance and value for understanding the deep network model's representation of visual cortex information and improving the capability of visual information analysis of neural signals.This paper aims at"expanding the dimensions of visual cortical information representation by using the deep network models for visual analysis".It combines with the advantages of the deep neural network models in computer vision.From the information representation features such as"multitasking","spatial connection"and"consecutive semantic representation"of the visual cortex,this paper constructs deep network models and explores new fMRI visual information analysis methods.The main work is as follows:1.Deep network model of multi-task information representation of visual cortex for visual encoding.At present,CNNs for research on visual encoding based on fMRI are mostly limited to classification task.In order to explore the impact of CNNs driven by other tasks on encoding performance,based on the task diversity representation of voxels in visual cortex,this paper constructs deep network model of multi-task information representation for visual encoding by using CNNs driven by different tasks.This paper extracts the features of natural images based on pre-trained classification network and segmentation network,linearly maps the classification features and segmentation features to the voxel responses,and realizes the construction of encoding model.The experimental results show that the encoding model based on segmentation network shows better performance on 35.05%of voxels,and the model based on classification network has better performance on 64.95%of voxels,which shows the diversity of visual expression in human brain and provides a basis for the similarities and differences between feature expression of CNNs and visual processing mechanism of human brain.2.Graph convolutional network model of connection information representation of visual cortex for visual classification.The visual system of human brain is a complex network structure,and the connection relationship between visual areas plays an important role in the expression of visual information.However,such information has not been effectively expressed in the existing visual classification models.This paper introduces the connection relationship between voxels in visual cortices and constructs a graph convolutional network?GCN?model used for fMRI visual classification.Firstly,based on the anatomical connection of the visual areas,we convert fMRI data into graph data,and then use GCN with three convolution layers to establish an end-to-end visual classification model to achieve category decoding.The experimental results show that the classification performance of graph data based on anatomical structure is significantly better than those based on no connection,random connection and internal connection in visual areas(two-sample T-test,and the P values are 6.96×10^-6,0.0017 and 0.0135,respectively.).The analysis shows that the model can effectively use the representation of connection information of visual cortex,and also illustrates the potential of applying the representation of connection information to the analysis of brain visual data.3.Conditional generative adversarial network model of semantic representation of visual cortex for visual reconstruction.After the visual cortex of human brain receives external stimuli,the stimuli will be processed to form an"understanding"in visual cortex.Visual cortex will generate various information representations from low-level to high-level features of the stimuli.However,how to effectively extract and use high-level semantic information to reconstruct stimuli is still a difficult problem.This paper regards the voxel responses as high-level language for the brain to process external visual stimuli.Based on this semantic prior,the Conditional Generative Adversarial Network?CGAN?for generating images from text is introduced to construct a model of semantic information representation of visual cortex for reconstruction.In the CGAN model constructed in this paper,the generative model maps the preprocessed voxel responses to the semantic description space.The semantic vectors are used as conditional priors and concatenated to the noise vectors to generate reconstructed images;the discriminative model distinguishes whether the input image is a natural image or a generated image and determine whether the image matches the responses.Experimental results show that the model constructed in this paper can reconstruct visual stimuli from the fMRI responses in an end-to-end manner.The average accuracy of subjective evaluation is 75.88%,and the average accuracy of objective evaluation is 68%.