Research On The Relationship Between Gesture And Speech Information For Human Computer Interaction

One of the most important research fields in computer science is human computer interaction (HCI). A lot of researches and projects have been carried out for years in this area such as speech recognition,speech synthesis,facial expresstion recognition,gesture recognition and gesture synthesis.But all of them are modality-specific. In contrast to the human world, the interaction between human and human is a very complex procedure. The information is conveyed by multi-modality. In order to achieve the same naturalness as human being's, the relationship between multi-modality must be studied. The goal of this paper tries to find out what the relationship between the two modalities of speech and gesture. This research was funded by national 863 project"The research on new methods for spoken language generation in natural HCI"(No.2006AA01Z138).We record 4 video and audio clips from television interview program and the linguistic and gesture information is annotated based on our multimodal coding scheme. The statistic analysis is made and the results support the Interface Hypothesis for speech and gesture production.This paper is organized into three parts. The first part introduces the multi-modal coding scheme and linguistic annotation convention. The second part includes a perceptual experiment, to find the role that the speech and gesture imformation that contributes to the language comprehension. In the last part, we focus on studing the relationship between the speech stress and hand movements, the speech stress between the hand movements and head movements, the relationship between hand movements and prosodic boundary.