Cross-Modal Visual Reconstruction Of FMRI Based On Self-supervised Learning And Vector Quantization

Understanding the structure and function of the brain is one of the most challenging scientific issues in this century.In-depth exploration of various functions of the human brain at different scales and levels not only has great significance for the diagnosis and diseases treatment,but also can effectively promote the development of brain-like intelligence and brain-computer interface technology.Research that attempts to use neuroimaging data to analyze the mechanism and function of the brain is called neural information encoding and decoding,decoding the visual information from the neural response of the brain is the first attempt to "read brain".Visual stimulation will induce specific neural activity in the visual area of the brain,these neural activities can be captured by functional magnetic resonance imaging(fMRI)and other neuroimaging methods.The purpose of this paper is to reconstruct the visual image more accurately from the neural image data through the latest machine learning technology,so as to promote the development of brain-like intelligence and braincomputer interface technology.The main research contents and contributions of this paper are summarized as follows:1.The research status of neural information decoding are summarized,the machine learning and deep learning algorithms used in neural decoding tasks are introduced,and the design ideas and development process of cross-modal visual reconstruction algorithm are introduced in detail.2.A self-supervised cross-modal generation model is proposed.Due to the highly complexity of human brain,the small sample size of neural image data,the high dimensionality,the diverse modalities and the low signal-to-noise ratio,it is difficult for common deep network model to learn the precise mapping relationship between the two modalities.We use self-supervised learning to make full use of limited paired samples to mine the rich hidden information in the two modal data,thereby effectively reducing the over-fitting of the network,increasing the generalization performance of the model,and enabling the model to learn accurate mapping relationship between two hidden spaces.Experimental results on multiple public datasets show that the method can precisely reconstruct binary contrast images,handwritten numbers and characters from brain responses.3.A cross-modal generation model based on vector quantization is proposed.Since the natural image contains both high-level semantic information and rich texture,contour,color and other structural information,It is difficult to accurately model the mapping relationship between the implicit representation of natural images and brain neural activities,so the conventional neural image decoding methods mostly try to reconstruct the visual image on simple character image datasets.To deal with above problems,this paper combines the ideas of vector quantization technology,and proposes a cross-modal visual generation model based on self-supervision and vector quantization.The model can extract the hidden representation of complex natural visual images and establish an accurate relationship with the hidden representation of neural activities.Experimental results show that the proposed method has a significant improvement over the previous method in reconstructing complex natural images.