Cognitive Process Of Audiovisual Modality Shift Effect And Modality Transform Based Visual Guidance System

In our daily life, human beings acquire information from several kinds of sensory modelities all the time. In order to accomplish different tasks, we need switch our attention among different sensory channels. Further, when a sensory system is impaired, we hope to substitute it with an alternative sensory channel. This thesis presents two studies on audioviusal modality shift effect and transform:The first part of the thesis is to investigate the cognitive process of modality shift between vision and audition. We studied the behavior and electrophysiological characteristics of this top-down attention switch from auditory channel to visual channel, or vice versus. With the high temporal resolution of event-related potential (ERP), we calculated the time cost of audiovisual attention switch according to the latency changes in the specific ERP components. Our results showed that attention switch from vision to audition cost about 152ms, while attention switch from audition to vision was about 96ms. In the ERP waveforms, N1 and P2 components evoked by auditory cues were significantly larger than by visual cues. The attention switch from visual channel to auditory channel evoked a negative component (less than 2μV) in frontal cortex and central region, and the ERP signal for maintenance of visual attention was closed to 0μV. However, the auditory attention maintenance evoked a negative component (less than 2μV) in frontal cortex and central region while ERP amplitude for attention switch from auditory channel to visual channel was closed to 0μV. Since the amplitude of ERP wave represents the cost of cognitive resources, the above-mentioned results were in line with the 'visual dominance' theory. Modality shift also caused an increase of amplitudes of N1 and P2 for the auditory stimulus, and a decrease of amplitudes of N1 and P2 for the visual stimulus. We assume this effect was related to the automatically attention-capturing characteristic of auditory stimulus and the low visual attention level after modality shift.The second part of this thesis is to develop a blind guidance system SoundView based on auditory substitution of vision. In detail, we employed embedded system to recognize important objects in the environment and encode them into corresponding speech signal to substitute the lost vision of the blind. SoundView captured video stream through the camera and implemented image processing algorithm to recognize important objects. According to the location of the objects, SoundView built 3D speech signal. The user could recognize objects from the speech and locate the objects with the stereo information. To avoid the complicated pattern recognition, all interested objects in the environment were tagged with barcodes with the detail information of the objects in an embedded database. In order to optimize SounView with wider range and high speed, we used a telephoto lens and improved image processing strategies for the barcodes in different distances. Also, we optimized the image processing algorithm and substitute some of the functions with library functions to save processing time, which greatly increased the processing speed. As a result, SoundView could recognize barcodes at the speed of 12frames/s effectively in the range of 0.5m-6.5m.