Research On Brain-Controlled Digital And Character Spelling System Induced By Audio-Visual Hybrid Stimulation

Brain-computer interface is a new technology that can directly control the operation of computer or other machine by using its EEG signal without the human neuromuscular,so as to realize the information interaction between human brain and external equipment.Aiming at the problem of low coding efficiency and instruction recognition accuracy in current brain-computer interface technology,this thesis studies the induction paradigm and instruction information decoding of brain-controlled character spelling system and verifies the effectiveness of the proposed audio-visual hybrid stimulation induction brain-computer interface.Firstly,in order to study the effect of visual and auditory stimuli and their semantic matching on the performance of brain-computer interface,four stimulation modes of digital spelling experiments were designed,which were single visual stimuli,single auditory stimuli,semantic consistent audiovisual stimuli and semantic inconsistent audiovisual stimuli.Through statistical analysis,the behavioral characteristics,event-related potential characteristics and off-line classification accuracy of different stimuli were compared.It is proved that semantically consistent audio-visual stimulation is a kind of stimulation that makes the brain respond faster,stronger and the recognition accuracy is higher.In this thesis,we propose a novel method for extracting EEG features in space-frequency domain.We use common space mode to filter EEG signals in space and decompose the filtered signals into 5 different frequency bands and find their average power spectral density as features.After that,support vector machine algorithm was used to the offline analyze of the semantic consistent digital spelling experimental data,and the classification accuracy of all subjects could reach more than 95%.Secondly,a brain-controlled spelling paradigm with 40 output characters is designed through semantic consistent audio-visual stimulation.The experimental paradigm is based on two-stage coding,group code + character code.By means of a single visual stimulus,the target character group is encoded by code division multiple access.In the second stage,semantically consistent audio-visual stimuli were used to encode the target characters.Based on the above paradigm,the brain-controlled character spelling experiment was carried out and the corresponding EEG signals were collected.The event-related potential characteristics of the target stimulus were analyzed.Based on the idea that the objective function with the smallest intra-class variance and the largest inter-class variance was used to replace the logarithmic loss function,a deep linear discriminant analysis method based on convolutional neural network was proposed for the classification and recognition of spelling instructions.The off-line experimental results showed that the recognition accuracy of the proposed method was improved compared to that of convolutional neural network and 3D convolutional neural network methods,with the highest up to 75.8%(stage ?)and 88%(stage ?).In summary,this thesis,taking the application of brain-controlled spelling as an example,designs and implements the experimental paradigm of audio-visual hybrid evoked character spelling and proves its validity.Furthermore,a new method of feature extraction and deep learning recognition for brain-controlled character spelling instruction is proposed.The research results have certain reference value to promote the practicality of the brain-controlled character spelling system.