Chinese Sign Language Synthesis Driven By Speech And Text

To help deaf people integrate into normal community better and improve theirweak position, a growing number of researchers are involved in the research of signlanguage. In recent years, a method by synthesizing sign language animation has beenproposed to facilitate the deaf understand and receive information through watchinganimations. To get more realistic and intelligibility sign language animation, prosodicinformation in sign language is necessary. As the same important role in theexpression of speech, prosody information can improve the expression ability of signlanguage and can provide additional function to help understand the content which thegesture maker wants to convey. When synthesizing of sign language animation,semantic text is not the only information which provides the necessary information forthe expression of sign language, but also it’s necessary to provide the prosodyinformation used for expressing the desired information in sign language. This articlestarts research from prosodic information in voice, and considers using the richprosodic information in the phonetic representation, and then maps the prosodyinformation to sign language prosodic parameters, at last sign language animationwith prosodic information can be got.In this paper, in order to extract the prosodic information in voice, basic acousticparameters such as syllable duration, sound intensity and pitch are chosen based onthe characteristics of Chinese prosody. Syllable duration is got by a method usingendpoint detection based on the existence of vowel in Chinese, then each syllableboundary can be divided; sound intensity is the short-time energy of every syllable;for the calculation of the pitch, it is estimated by using the cepstrum parameters. Inthis paper, based on these basic acoustic parameters, a feature vector is constructedout for identifying the emphasis pattern of prosody, and hence the hidden Markovmodels (HMM) can be trained. In order to improve the robustness of the models, aseries of relative values are used for forming the vector, it’s due to the process of thehuman ear perceiving stress. Prosody information can be got through the HMMs, andafter mapping the phonetic prosody to sign language prosody parameters it can bedirectly used to synthesize Chinese sign language animation. The mapping of prosodyparameters can be achieved by Chinese Sign Language Markup Language (CSLML)used for describing signs and prosody. Finally, the paper implements a system at theset-top box platform to achieve an accompanying television programs while playingChinese sign language animation that can effectively help deaf people watch TVshows and receive social information.