| The ultimate goal of Artificial Intelligence is to give machines human intelligence,Studying and simulating the brain is the only way to achieve this.Presenting the neural activity state of the brain depending on biomedical technology,decoding the brain ideology through non-invasive brain-machine intelligent technology,and making machines have self-awareness is a crucial research direction in the field of artificial intelligence in the future."Brain science and brain-like research" have been included in the national 13 th and 14 th Five-year Plan,and in 2021,the Science and Technology Innovation 2030 project application guidelines issued by the Ministry of Science and Technology also deployed brain science and brain-like research tasks from various aspects,which clarified the importance and urgency of brain-like computing and brainmachine intelligence research.The human brain has similar ability to represent the same kind of visual stimuli,and the Electroencephalogram(EEG)characteristics can reflect the content of the image that arouses the original EEG signal.Aiming at the needs of national strategic planning and scientific research,this paper studies the correlation between visual stimuli and EEG signals.The research is also conducted based on 40-category EEG signals evoked by images,so as to establish the mapping between human visual information and external visual stimuli.On the one hand,an image scene understanding model based on human visual cognition is developed.The human visual function is simulated by using machine intelligence,to form a brain-computer hybrid enhanced intelligence based on cognitive computing.On the other hand,EEG signals and images used to arouse EEG signals are modeled simultaneously to explore a cognitive learning method based on closed-loop generative-adversarial cognitive learning methods.When monitoring the EEG signals,the computer can generate images corresponding to the things that the brain pays attention to,so as to realize the hybrid enhanced intelligence of human-machine collaborative cognitive system in the human loop.The main research work and innovative results of this paper are as follows:(1)EEG decoding and classification based on the characteristics of time-space domain: Considering the spatial correlation and temporal inherent characteristics of multi-channel signals,the Spatial-Temporal Network,which is composed of Convolution Neural Network(CNN)and Long Short-Term Memory Network(LSTM),is designed in this paper,so as to extract the visual-related underlying features of EEG.The convolution network conducts EEG space domain analysis in channel level and explores the linkage relation between each channel.Then the LSTM network conducts time-domain analysis of signals,so as to produce explainable features of neurophysiology,finally encoding them into a descriptive signal representation containing specific content,establishing the mapping between human visual information and external visual stimuli,and realizing the accurate decoding of visual stimulation EEG signals.The distinguishing degree of the extracted features to EEG signals reaches 96.47%.This method utilizes both the ability of CNN to identify spatial features and the ability of LSTM to extract sequence features.Compared with other EEG representation methods,the classification accuracy has been significantly improved.It provides a theoretical basis for subsequent image scene understanding based on human visual characteristics and high-quality generation of visual stimulus images.(2)The model on image scene understanding based on EEG-visual characteristics:In view of lacking biological basis of current scene understanding methods,this paper maps human visual perception ability to machine.Firstly,an EEG signal analysis method LSTMS-B combining deep learning and ensemble learning is proposed,which improves the expression ability of EEG features to EEG signals through multiple network co-decision.Then,EEG features are applied to automatic image classification.The Res Net-KNN regression model was trained using EEG features and stimulus images.The image is mapped to the low-dimensional features of the EEG signals,making the machine learn the visual representation of the brain,and then predict the estimated EEG characteristics of the images that are not involved in the EEG acquisition,so that the human visual perception ability is mapped to the machine and the machine can predict the classification of images(visual scenes)based on human visual ability.During the test procedure,49,429 image samples in Image Net dataset are used.It proves the correctness of the idea of image classification under the EEG signals source which realizes the image scene understanding based on the human visual cognitive characteristics,and 90.72% weighted average classification F1 score of the images is obtained.An image scene understanding model which can process massive information based on human visual characteristics is realized.(3)Research on generative adversarial reconstruction method based on EEG visual characteristics: Aiming at the problem that the collaborative representation ability of human-machine cooperative visual cognition system in human loop is not strong,a method Bi-LSTM-Att GW is proposed to decode EGG signals,in which the attention mechanism is added into the bidirectional long short-term memory network by means of attention gate and attention weighting.And then,the signal coding feature is used as the condition of conditional Spectrum Normalized Generative Adversarial Network(SNGAN),so as to constrain the image generation process.When the original EEG signals are recorded,an image,corresponding to the thing that the brain pays attention to in a specific scene,is generated.In the test of image generation,the conditional SNGAN can generate images according to the EEG signals evoked by visual stimuli,and the average Inception Score of generated classes images is 7.17.In terms of subjective vision,the methods used also produce higher quality images that better fit the label.It proves that this method can realize the transformation of "Image-EEGImage".It is helpful to improve the collaborative representation ability of humanmachine collaborative visual cognitive system.Decoding human brain activities related to visual scenes has a great impact on brain-inspired computing and brain-computer interaction fields.This paper is aimed at dealing with EEG analysis and studying the correlation of EEG signal related with human vision,it is proved that the brain neural activity data can provide more abundant supervision for deep learning models.On the one hand,visual classification methods consistent with human neural data are produced,which can be used to perform brainmachine hybrid-augmented intelligence research tasks and realize the image scene understanding based on the human visual cognitive characteristics.One the other hand,the images reconstruction guided by EEG signals are completed.It improves the cooperative representation ability of human-machine cooperative visual cognition system in human loop.The study can provide theoretical and technical support for deep neural network-based EEG-visual decoding methods,and provide meaningful insights for further study of human visual perception system. |
