A Study On Cognitive Mechanism About Audio-visual Integration Of Polyphone Using Multi-variate Pattern Analysis

Several event related potential(ERP)components have been observed in the study of language cognition based on the method of superposition.However,little is known about what the functional connectivity were in the process of polyphones cognition in semantic integration of audio-visual information.In the present study,we aimed to investigate the functional interaction in the brain during the cross-modal integration of polyphonic characters in Chinese audio-visual sentences.The visual sentences were all semantically reasonable and the pronunciations of the polyphonic characters in corresponding sentences contexts varied in four conditions.To measure the functional connectivity,correlation,coherence and phase synchronization index(PSI)were used,and then multivariate pattern analysis(MVPA)was performed to detect the consensus functional connectivity patterns.And these analyses were confined in the time windows of the three related event-related potential(ERP)components of P200,N400 and late positive shift(LPS)to investigate the dynamic changes of the connectivity patterns at different cognitive stages.The results showed that significant functional connectivity patterns were obtained in the time windows of these components.When differentiating the polyphonic characters with abnormal pronunciations from that with the appreciate ones in audio-visual sentences,significant classification results were obtained based on the coherence in the time window of P200 component,the correlation in the time window of N400 component,and the coherence and PSI in the time window of LPS component.Further,a multi-kernel learning method based on the three functional connectivity indices found that significant classification accuracies were achieved during the three time windows of the related ERP components.Moreover,the spatial distributions in these time windows were also different,with the recruitment of frontal sites in the P200 time window,the frontal-central-parietal regions in the N400 time window and the central-parietal sites in the LPS time window.These findings demonstrated that the functional interaction mechanisms were different at different stages of audio-visual integration of polyphonic characters.