Research On Natural Image Retrieval With Brian-computer Interaction Based On Real Time FMRI

The human brain is the center of consciousness, the center of mind and the center of control. The rapid development of neuroimaging makes it possible to read brain signals, which greatly promotes the progress of neuroscience. Especially after the invention of functional magnetic resonance imaging(fMRI), function imaging spatial resolution of brain activities increases greatly, which makes it possible to accurately interpret both complex brain perceptual and cognitive neural activities.Since vision is the main source of human information, the interpretation of human visual system is becoming a research focus in brain science. Image retrieval based on fMRI can directly restore the visual information from brain activities. It is not only the most challenging and attractive technology at the present stage in the interpretation of visual information, but also plays a prominent role in the research of human visual information processing mechanism.To interpret the visual information of our human brain while receiving natural image stimulus, this thesis focuses on the visual encoding model that merges primary and intermediate visual features together, and the semantic classification method based on convolutional neural network encoding and decoding model. Based on these researches, an image retrieval system is built combining with the brain-computer interaction technology based on real time functional magnetic resonance imaging(rt-fMRI). The main work includes:1. A visual encoding model with intermediate visual features dictionaries is built, and an image identification method based on the model is implemented. Current visual encoding models often treat the early visual cortex as a whole, emphasize the representation ability of primary visual features, and neglect the effects of intermediate visual features. It would affect the accuracy of visual encoding model. This study constructs intermediate visual features dictionaries for both blob and corner features using sparse dictionary training. The constructed dictionaries are used to simulate the voxels of visual areas, and they have been combined with the Gabor Wavelet Pyramid(GWP) to construct the visual encoding model with primary and intermediate visual features. In this thesis, the image identification accuracies based fMRI have been got by the improving model, the results show that the optimized visual encoding model can effectively characterize the specific primary and intermediate visual features, and improve the accuracy of image identification.2. A semantic classification method based on convolutional neural network encoding model is proposed. Traditional semantic classification methods treat the voxels got by fMRI as individuals, the significant level of single voxel activation of visual stimuli has been the standards in feature selection for semantic classification. However, the voxel activation value cannot directly reflect how much visual semantic information the voxel contains, this leads to poor performance of the visual semantic interpretation. This study constructs encoding model of semantic related visual area, and uses feature semantic similarity to be a standard for feature selection. The selected voxels can contain more semantic information in a certain extent. The natural image classification experiment with four kinds of semantic has verified the method in the aspects of semantic classification.3. A brain-computer interaction strategy for rt-f MRI image retrieval is proposed, and a brain-computer interaction image retrieval system based on rt-fMRI is built. The system combines interpretation and identification algorithm of visual information, and can online retrieve similar images with visual stimuli from an image library through fMRI signal interpretation. At the same time, this study introduces a real-time feedback mechanism into the system, and designs a brain-computer interaction strategy based on online classification of visual attention states. The method can implement two-way communication of visual information between human and computer in order to improve the accuracy of visual semantic interpretation. The experimental results show that the system can accomplish the real-time image retrieval task effectively and stably. By introducing the brain-computer interaction module into the system, the classification accuracy increases significantly.