Research On Brain Response Of Auditory And Visual Competitive Information Under Different Semantic Correlations In Audiovisual Bimodal Brain-Computer Interface

Visual and auditory senses are the most important accesses for people to communicate with the world,so they have been widely studied in neuropsychology and brain-computer interface(BCI)researches.Previous studies have promoted on audiovisual integration effect to better enhance people’s perception of multisensory.Nevertheless,the competition mechanisms of audiovisual interaction in BCI are still needed.In this paper,we focused on the brain response under audiovisual competition paradigms with semantic relevant and semantic irrelevant stimuli.In this study,we combined auditory monosyllabic stimuli and two kinds of visual stimuli to design audiovisual relevant-disturbance paradigm(RD)and irrelevantdisturbance paradigm(ID).This study mainly analyzed responses under RD and ID conditions from three aspects: spatio-temporal feature of event-related potentials(ERP),support vector machine(SVM)classification and brain network analyses.The main results were as follows:1.The spatial and temporal analysis method of ERP found that the amplitudes of ID paradigms were higher than RD paradigms,indicated that relevant-disturbance paradigms in audiovisual interactions demand more cognitive resources.Compared with auditory-attention paradigms,visual-attention paradigms elicited significantly higher ERP amplitude,which means V-attention paradigms are more likely to elicit stronger ERP response with the distraction of other modalities.In addition,we found that long ISI causes higher ERP amplitude and longer period.2.SVM was used to classify the target stimulus and non-target stimulus of each paradigm,the results showed that the accuracy of V-attention tasks was significantly higher than that of auditory attention tasks,suggested that the visual modality occupies the dominant position in the audiovisual bimodal situation.However,semantic correlations had no significant effect on classification results,which illustrated that classification accuracy is not necessarily proportional to amplitude of event-related potential.3.The results of partial directed coherence brain network analysis showed that semantic correlations between audio and visual stimuli had significant effect on the brain network connectivity in the short ISI.Compared with long ISI,short ISI paradigms activated more regions especially the temporo-occipital location in the brain.It revealed the fact that the response of fast stimulation in audiovisual competition paradigms requires more cognitive resources.In conclusion,this study aims to explore the differences in audiovisual cognition with different competition conditions.Two kinds of audiovisual competition paradigms were designed in this study.It was found that semantic relevance and audiovisual attention channel had significant effects on brain cognitive information processing and classification performance.The results help to understand and explain the differences between neuropsychology and brain-computer interface research on bimodal stimulation,and provide theoretical support for improving the efficiency of BCI.