A Study On Human-computer Interaction Technology Based On Bioelectrical Signal And Its Application In Virtual Reality

Human-computer interaction(HCI)refers to the process of information exchange between human and computers or other external devices.HCI technology based on the bioelectrical signal is a kind of new interaction technology which is different from the traditional mouse and keyboard.It can translate the bioelectrical signals into control command for computer directly by algorithm,without any intervention of hands and feet,which can provide disabled people's daily life with a kind of effective auxiliary.Virtual reality(VR),as the most advanced computer simulation technology,is an excellent platform for human-computer interaction.This thesis proposes several practical HCI systems which combine the bioelectrical signal with virtual reality technology;study the HCI technology based on bioelectrical signal(mainly Electroencephalography(EEG)signal and Electrooculography(EOG)signal)and its application in virtual reality.Firstly,this thesis introduces three-dimensional(3D)modeling and animation technology of virtual reality into the EEG-based HCI paradigm;proposes a 3D P300 speller based on stereo visual stimuli.EEG-based HCI is also known as brain-computer interface(BCI).P300 speller is the most popular and widely concerned application type of BCI.This thesis proposes a new 3D paradigm to improve the performance of traditional P300 speller.In this system,we use 3D flashing cubes instead of two-dimensional(2D)flashing planes to build our 3D paradigm,and each stimulus is composed of a fast flash(color change)and a 3D motion.Twelve healthy subjects took part in an offline experiment and an online experiment.Experimental results show that the proposed 3D paradigm can elicit higher amplitudes of P300 waveforms than the traditional 2D paradigm,thus significantly improving the classification accuracy and information transfer rate.Secondly,based on the superior performance of 3D BCI in the aspect of evoking ERP obtained in the first work,this thesis proposes a new 3D BCI system for the assessment of visual startle in patients with disorders of consciousness.Previous results have confirmed that BCI technology can effectively support the assessment of auditory startle,visual localization and communication function in CRS-R scale.However,the traditional 2D BCI paradigm is difficult to simulate the dynamic effect of finger prodding in the assessment of visual startle.At present,the research in this field has been in a blank state.In view of this situation,this thesis designed a 3D BCI system for assisting visual startle assessment of patients with disorders of consciousness based on virtual reality technology.In this system,standard stimulus and deviant stimulus with a ratio of 4:1 were used to form the stimulus sequence,in which the standard stimulus used a background picture of a static finger and the deviant stimulus used a dynamic animation of a finger poking to eye to induce the P300 potential of the subjects.Eight patients with disorders of consciousness participated in the assessment of BCI and CRS-R scale.Detailed data analysis shows the effectiveness of BCI system in detecting visual startle response.Thirdly,considering that EOG signal has advantages of larger amplitude,higher signal-to-noise ratio and better stability than EEG signal,a virtual reality interactive system based on EOG is proposed.Different from the existing VR systems based on EEG or EOG,this system uses the synchronization between the blink response time of human eyes and the flash of buttons to distinguish the target button from the non-target button.The users can issue the corresponding control command by executing the blink that is synchronized with the flash of the target button.In this project,we integrated the small bioelectrical signal amplifier,flexible dry electrode,mobile terminal and VR headset,giving full play to the advantages of EOG signal acquisition in portability,and realized the mobile portable VR application based on EOG.This system takes indoor environment control simulation as a test case,and ten healthy subjects participate in the online VR interaction experiment.The experimental results show that all subjects can use the system to efficiently complete various interactive tasks,and the system's multiple performance indices are better than the existing EEG-or EOG-based virtual reality interactive system.Finally,considering the characteristics of EEG and EOG,this thesis proposes a virtual reality interactive system based on hybrid brain-computer interface.In the previous work,we have proposed a VR system based on EOG,which achieved better performance indices than the existing EEG-based system.However,that interactive system can only provide the user with discrete control commands,while unable to output continuous control commands.So,a new system is designed by improving that system.In this study,EOG and EEG algorithm are fused: EOG signal is used to output discrete control commands,while EEG signal is used to output continuous control commands.In this way,the interactive function and performance of the BCI-VR system can be effectively improved.In addition,we have built a multi-channel large-screen immersive BCI-VR platform,on which,according to the above interactive methods,a new BCI wheelchair control paradigm was designed by taking outdoor wheelchair control simulation as a test case.Five healthy subjects participated in two online experiments.The results have demonstrated the effectiveness and reliability of this BCI-VR system.