Cross-modality Effect In Implicit Learning Of Temporal Sequences

Implicit learning refers to the process in which individuals effortlessly acquire the structural relationships between objects without conscious awareness.Temporal sequences,characterized by chronological order,are common in daily life.Acquiring knowledge of temporal sequences aids people in better behaviour organisation and environment adaptation.However,obtaining temporal information in an entirely isolated environment is not available and depends on the input of sensory modalities such as vision or audition.Therefore,switching and integrating temporal information across different modalities play important roles in enhancing ability to cope with multifaceted tasks more efficiently.However,previous studies investigated the formation of temporal information representation during the learning process using a two-stage experimental framework called "learning-testing" with cross-modality transfer.Although this approach enables us to explore whether temporal information can be transferred across modalities,it is challenging to determine whether the transfer process is accompanied by a gain or a loss in learning effects,resulting in clarifying difficulty whether the encoding of temporal information is bi-directional or asymmetric between the visual and auditory modalities.Therefore,this dissertation adopts a cross-modality switching approach during the learning process to eliminate the interference of the delayed testing effect in cross-modality transfer.The approach aims to investigate the learning effect of temporal information before and after cross-modality switching,resulting in providing a more comprehensive understanding of the representation patterns of temporal sequence information during implicit learning.Moreover,by disrupting the encoding of temporal sequence information in the non-learning modality during implicit learning,it is possible to further validate the roles of visual and auditory encoding in processing temporal representations.In conclusion,exploring the cross-modality effect on temporal sequence during implicit learning,along with its associated neural oscillatory signaling-directed switching,contributes significantly to enhancing our comprehension of how the brain effectively handles temporal sequences during implicit learning.The present dissertation adopts a temporal-sequence-based serial reaction time task(t SRT),combined with EEG(Electroencephalography)techniques and Phase Transfer Entropy(PTE)analysis,to systematically investigate the temporal-sequencebased implicit learning process.This dissertation investigates the behavioral patterns of cross-modality effects during implicit learning of temporal sequences and the underlying neural oscillatory mechanism and further confirms the role of visual and auditory coding in the implicit learning of temporal sequences.This dissertation study was conducted in three studies containing six experiments.Study 1(Experiment 1 and Experiment 2)aimed to examine the pattern of learning effects of temporal sequence implicit learning in switching between visual and auditory modalities,and the effect of switching timing on modality switching effects with behavioral experiments.Study 1 used a t SRT task based on the temporal structure of repeated sequences.It was found that there were asymmetric modality switching effects during temporal sequence implicit learning,specifically,when temporal sequence implicit learning was switched from the visual modality to the auditory modality,there was a significant gain in implicit learning effects on the auditory modality and the presence of structural knowledge on the auditory modality was demonstrated.No such modality-switching effect was observed when switching from the auditory to the visual modality.In other words,there was no significant increase in implicit learning effects on the visual modality after the modality switch,and no significant structural knowledge was demonstrated.Also,the results of Study 1 found that the gain effect of visual-to-auditory modality switching only occurred when modality switching was performed in the middle of the learning period(Phase 4),while modality switching in the early learning period(Phase 2)did not lead to this effect,suggesting that temporal sequence implicit learning requires a certain amount of knowledge accumulation for audiovisual channel switching to occur.Study 2(Experiments 3 and 4)aimed to examine the effects of different audiovisual modality switching directions on the implicit learning effects of temporal sequences and the associated neural oscillatory signaling patterns.Study 2 also used the t SRT task combined with high temporal resolution EEG techniques and phase transfer entropy(PTE)analysis.Consistent with the behavioral results of Study 1,an asymmetry in the visual-auditory modality switching on the implicit learning effect of temporal sequences was demonstrated.The EEG results further revealed that temporally sequence during implicit learning evoked stronger theta neural oscillation energies compared to random sequences.Most importantly,asymmetric signaling patterns were also observed at the neural oscillation signal transfer,as evidenced by the fact that during learning in the visual modality,theta oscillations of the regular sequence were transferred from the auditory region to the visual region,while this signaling was not observed after switching to the auditory modality;during learning in the auditory modality,theta neural directional information transfer of theta oscillations was not observed between auditory and visual areas during learning in the auditory modality,whereas after switching to the visual modality,directional transfer of theta oscillation signals from auditory to visual areas occurred.Furthermore,stepwise linear regression model analysis revealed a relationship between the strength of theta oscillatory signal transmission and learning effects.It was found that only when switching from the visual to the auditory modality,the strength of theta oscillatory signaling from the auditory region to the visual region before the switching positively predicted the gain magnitude of the learning effect after the modality switching.These results further suggested that there is an associated asymmetry in the transmission of theta oscillation signaling behind the non-contrastive claim of modality switching effects at the behavioral level.Study 3(Experiment 5 and Experiment 6)interfered with the temporal information encoding on the visual and auditory modalities respectively to examine the effect of different modalities interference on the temporal sequence implicit learning effect and the related neural oscillation signal activity pattern,and further,verify the role of visual and auditory encoding in the process of temporal sequences representation.The results showed that when interference information was presented on the auditory modality,i.e.the interference information was auditory coding,although the subjects implicitly acquired visual temporal sequences knowledge,their learning effect was significantly lower than that of the no interference group.Moreover,the power of theta oscillations induced by the compliance with the temporal sequences rule was significantly lower compared to the no-interference group.At the same time,there was no longer a significant signal transfer of theta oscillations between temporal and occipital areas.In contrast,the implicit learning effect of auditory temporal sequences,the power of neural oscillations,and neural signal transfer were not affected when interference information was presented on the visual modality,i.e.when the interference information was visually encoded.These results demonstrated the important role of auditory coding in the implicit learning process of temporal sequences.In summary,this dissertation explored the cross-modality effects of temporal sequences implicit learning between the audiovisual modality and the underlying neural oscillation transmission patterns.The results showed that there was an asymmetry in the gain of learning effects due to modality switching during temporal sequence implicit learning,mainly in the auditory modality when switching from the visual modality to the auditory modality;At the neural oscillatory signal transfer,there is also an asymmetry in temporal sequence implicit learning in the audiovisual modality.Specifically,it was found that only before the modality switch in the visual modality,the strength of theta oscillatory signaling from the auditory region to the visual region significantly and positively predicted the gain magnitude of the learning effect after the modality switching.There is a specific relationship between visual temporal sequence implicit learning and changes in information directional transmission.There was a specific correspondence between visual temporal sequence implicit learning and changes in information directional transmission.Furthermore,the effect of modality interference on temporal sequence implicit learning was asymmetric,with interference in the auditory modality not only affecting the behavioral performance and theta oscillatory in visual temporal sequence implicit learning but also leading to impaired transmission of theta oscillatory signals from the auditory to the visual domain.To sum up,these findings supported the conclusion that auditory neural encoding plays a fundamental role in the implicit learning of temporal sequences.Therefore,compared to visual encoding,auditory encoding demonstrated a distinct advantage in the processing of temporal sequence information,thus indicating an inherent asymmetry in the manner by which temporal sequences are processed between the visual and auditory modalities.The findings presented within this dissertation shed light on the underlying mechanisms and processes through which the brain comprehends temporal sequences during implicit learning,thereby offering valuable insights for a comprehensive understanding of how temporal sequence information is processed within the brain.