| Everyday experiences provide us with a multitude of information sources and opportunities for action. To optimize our behavior in the future we learn to associate perceived multisensory information and self-performed actions with perceived outcomes of our actions. Perception, learning, and memory are therefore multisensory and include information related to physical actions. However, much is left unknown about the ways in which multisensory and motor information modulate later behavior and information processing. To address this, the current experiments utilized behavioral and functional magnetic resonance imaging (fMRI) measures in order to better understand the impact of multisensory and active motor learning on subsequent perception and memory at both behavioral and neural levels. In Experiment 1, participants were presented with cue-target associations comprised of pairs of nonsense objects, pairs of nonsense sounds, objects paired with sounds, and sounds paired with objects. Subsequent to this, successful within-modal recall was associated with modality-specific reactivation in primary perceptual regions, and was more accurate than cross-modal retrieval. When auditory targets were correctly or incorrectly recalled using a cross-modal visual cue, there was reactivation in auditory association cortex, and recall of information from cross-modal associations activated the hippocampus to a greater degree than within-modal associations. In Experiments 2 and 3, participants learned visuo-auditory-motor associations between novel objects and the sounds they produce, either through self-generated actions on/with the objects (active learning), or by observing an experimenter produce the actions (passive learning). Active learning was faster and led to more accurate recognition of audiovisual associations than passive learning. In motor and haptic regions there was greater activation during both the perception and recognition of actively learned associations, and functional connectivity between visual and motor-related processing regions was enhanced during the presentation of actively learned audiovisual associations. In addition, several regions showed greater multisensory gain after active learning compared to passive learning. The current experiments provide new evidence that multisensory and active learning impact associative perception and recognition at both the behavioral and neural levels. Together these findings not only extend previous empirical work but also have important implications for theories of the relationship between perception, action, and memory. |
