Sequential Neural Processes In Visuo-tactile Crossmodal Paired-associate Learning

Studies have indicated that a cortical sensory system is able to process information from different sensory modalities. However, it still remains unknown when and how a cortical system integrates and retains information across sensory modalities during learning. In this dissertation, the neural mechanisms underlying visuo-tactile crossmodal associations and memory were investigated in several experiments by using electroencephalographic (EEG) and behavioral approaches.In the first part of this dissertation, the history and current progresses in the fields of crossmodal working memory and paired-associate (PA) learning were reviewed. Based on this review, the specific aims and hypotheses of this study were proposed.The second part is the main body of the stduy that consists of four experiments.1. In Experiment 1, a delayed PA learning paradigm and EEG recordings were used to address learning-related ERP (event-related potential) components. During the PA task, subjects were required to identify predefined visuo-tactile (crossmodal) or visuo-visual (unimodal) pairs through a process of trials and errors. Feedback was provided during learning phases, but not during learned phases. Amplitudes of a certain ERP component were compared between learning and learned phases (referred to as learning effects). In N400/P400 complex, the N400 ERP component obtained from frontal recording sites between 300-600 ms after the onset of the first stimulus showed learning (but not modal) effects, suggesting that this component represents neural activity during associative learning, regardless of sensory modality. By contrast, the P400 component in central sites showed both learning and modal effects, indicating that P400 represents neural activity during associative learning, depending on sensory modality. A late posterior negative slow wave (LPN) was observed between 500-700 ms in posterior sites, which showed learning effects only in the crossmodal task. This component may therefore only reflect neural processes of crossmodal associative learning. Furthermore, alpha-band responses (8-12Hz) were observed between 800-1500 ms of the delay period of the crossmodal task in both frontal and posterior sites, suggesting they represent neural activity of crossmodal working memory.2. In Experiment 2, the subject’s familiarity with visual stimuli used in Experiment 1 was examined by using a visual recognition task. Behavioral results showed that there was no statistically significant difference in task performance between subjects who had learned two or four VV blocks, indicating that familiarity with visual stimuli was not a major factor in the present study.3. The effect of changes in length of the interval between S1 and S2 on behavioral PA learning was also investigated by delay-varied PA testing tasks in Experiment 3. When the interval between S1 and S2 was shortened to 400 ms (stimulus onset asynchrony, SOA:400 ms; the onset of S2 prior to N400/P400), accuracy in both VT and VV tasks showed significant decline compared with that under the 1300 ms-delay condition. Correspondingly, longer time was needed for the subjects to indicate the paired-associations. However, when S2 was presented after N400/P400 but before LPN (SOA:600 ms), only VT task showed significantly longer reaction time.4. Further, EEG data in the delay-varied PA testing tasks were recorded and analyzed in Experiment 4. The modal effect was observed on three visually evoked early potentials:posterior N140, central P170 and posterior P220. These results suggest that visual-tactile cross-modal associations may affect visual sensory-perceptual processes in humans.In the third part, the main findings of the current study were concluded and generally discussed. The findings provide insights into the temporal dynamics (sequential temporal changes) of neural networks that mediate formation of crossmodal associations and working memory.