Chinese Sign Language Synthesis Based On Multi-Clues

As a kind of visual/gesture language, sign language(SL) delivers information through gestures and body movements. It’s the most natural and efficient way for the hearing impaired people to communicate with others. The ultimate purpose of sign language synthesis is achieving an accurate and universal way to display sign language and constructing a barrier-free environment for the hearing impaired people through which they can get access to information more conveniently and integrate into the mainstream society. With the development of 3D virtual human technology, sign language animation based on virtual human has become main research direction in the sign language synthesis domain. Limited by overall level of research in Chinese Sign Language and natural language linguistics process, sign language synthesis driven by text cannot extract non-gesture movements like facial expressions and blink of eye movements from pure-text input which prevent realistic and intelligible sign language animation. Therefore, how to synthesis more realistic and intelligible sign language animations has become a significant problem. Since facial expression contributes much for understanding sign language, including expressions in the synthesized sign language animation is necessary. This thesis studies Chinese sign language synthesis based on multi-clues, aiming at synthesizing sign language animation with facial expressions driven by multimedia information.This thesis proposes a method utilizing three types of input information which includes video, speech, and corresponding text to synthesize Chinese sign language animation. In order to gain facial expressions in sign language, video sequence of the input is used to extract facial feature points movements of the actor. Based on the MPEG-4 standard, facial animation parameters are calculated to drive the 3D virtual human face to move as the video face does. At the same time, gesture movements deduced from text are separate and sign language is a multi-modal behavior which means facial expressions should be coordinated with gestures. In order to achieve synchronizing behavior of virtual human, a speech-text alignment process is adopted to gain time information of each word, which is used to synchronize the expression and gesture. Finally, a prototype system is actualized in the set-up box environment where the input information includes video and audio.