Auditory And Visual Encoding Of Language And The Effects Of Language Experience

As a highly complex information processing task in human brain,language processing involves multiple stages,e.g.,the perception of physical modalities of language,lexical processing,and syntactic as well as semantic processing.This thesis focused on perceptual mechanisms of language modalities.The perceptual mechanisms are closely related to the characteristics of language modalities: During speech listening,the brain encodes the acoustic features of dynamic speech in real time,while during text reading,the brain flexibly adjusts reading time and order through eye movements.Thus,this thesis conducted a series of electroencephalography(EEG)experiments and eye movement experiments to study perceptual mechanisms of speech listening and text reading,respectively,and investigated the modulations of language experience on perceptual mechanisms.Additionally,based on ecologically valid passage comprehension tasks,computational models fitting the EEG responses and eye movements patterns were built to further the understanding of perceptual mechanisms underlying speech perception and text reading.The first study examined the neural encoding mechanism of speech envelope using EEG experiments.The study first investigated the computational mechanism of envelope-tracking responses.We tested the phase properties of two kinds of computational models,i.e.,linear time-invariant model and nonlinear oscillator model,with results showing that the linear time-invariant modeling matched the phase properties of recorded EEG responses.We further modeled the envelope tracking responses based on the linear time-invariant model,and investigated how language experience modulated envelope-tracking responses using between-group design in which two groups of listeners were recruited,i.e.,native listeners of the testing language and foreign listeners who do not understand the testing language.In addition,in order to test how the difficulty of speech perception affected the modulation of language experience,the signal-to-noise ratios of speech stimuli were manipulated.The study found that language experience can adaptively modulate envelope-tracking responses:under low-noise conditions,language experience impaired envelope-tracking responses;under high-noise conditions,language experience facilitated envelope-tracking responses.These results suggested that language experience can modulate the neural encoding of acoustic features during speech perception,and the modulation was depended on the difficulty of speech perception.The second study focused on the eye movement mechanism during questiondriven reading comprehension using a series of eye movement experiments.The study applied a task commonly used in language proficiency testing,i.e.,answering a specific question by reading a passage.The study recorded the reading time of each word using eye movement experiments and established a large-scale reading comprehension annotated dataset.As to computational modeling of eye movements,the study demonstrated that deep neural network(DNN)language models reached human-level performance on this task,and DNN attention weights account for human reading time.Further analyses showed that both DNN attention weights and human reading time were affected by the task.More importantly,during task optimization,DNN attention weights gradually approached the distribution of human reading time.On this basis,the study further investigated how language experience modulated human reading time by contrasting native speakers and second language speakers.Results showed that the reading time of native speakers and second language speakers were affected by the textual features,layout features,and task relevance.Additionally,the reading time on each word was shorter and less sensitive to layout features and question relevance for native speakers than second language speakers.These results indicated that language experience modulated the reading time,and the reading time of second language speakers was more strongly affected by reading goals.In sum,this thesis verified the linear time-invariant model can explain envelopetracking response,and proposed the DNN-based eye tracking model for question-driven passage reading.Additionally,this thesis revealed the modulation of language experience on envelope-tracking responses in speech perception and the reading time in question-driven reading.Furthermore,it is demonstrated that,without sufficient language experience,the brain engaged more active audio-visual processing to enhance the encoding of audio-visual information.With sufficient language experience,the language processing was more efficient,so the control of audio-visual processing was weakened.The methods and conclusions of the thesis can support the studies in language development and second language acquisition,and the computational models can potentially promote the researches on perceptual mechanism under natural scenes.