An ERP Study Of The Influence Of Iconic Gestures On Speech Comprehension

Face-to-face communication is essential in the process of interpersonal communication.While people are talking and laughing,they also make some corresponding body information,such as micro-expression,micro-action or some hand movements.This makes it necessary for people to integrate the information of each channel when understanding the content of the conversation.The main purpose of this study is to explore how the communicator integrates the information of different channels.What is the receiver’s reception mode for the information of the auditory and visual channels? Is it a "no-thinking" overall reception or a "chosen-admission" selective reception? In previous studies which are using English,Germany and other languages as materials show that effective gesture information can promote the processing and understanding of speech.However,using Chinese as an experimental material is rare.This study will explore the role of different gestures in speech understanding in the context of Chinese.To explore this issue,this study uses two experiments to explore the role of gestures in communication:Experiment 1 used 2(Gesture: Dominant,Subordinat e)× 2(Target Word:Dominant,Subordinate)of the design of the subject.The experimental process is to let the participants watch the video material with voice.The task is to try to understand the meaning of the video and record the EEG of 300~500ms after the target word.The results shows that under the four conditions,the N400 under the condition of gesturespeech consistency is significantly different from that under the condition of gesturespeech inconsistency.In communications,people always talk with not only generating iconic gestures,but also adding some non-realistic hand movements(Grooming),so experiment 2 uses 3(Gesture: Dominant,Subordinate,Grooming)×2(Target word: Dominant,Subordinate)to explore the effect of the iconic gesture on speech understanding after adding a certain proportion of grooming in Experiment 1.The experimental procedure was consistent with Experiment 1,and the EEG of 300 to 500 ms after the target word are recorded.It is found that the N400 is only significant under the condition of the subordinate gesture with subordinate target word.Combining the results of Experiment 1 and Experiment 2,the study draws the following conclusions:(1)Iconic gestures can promote understanding of voice information in people’s communication.(2)Grooming can reduce the integration of iconic gestures and speech.And integration is not a series of automated processes,but is subject to other external information.