Continuous Ultrasound Based Articulatory Movement Synthesis From Speech

Speech signal processing is one of the most popular research field in Computer Science research and engineering. Although the basic use of the automatic speech recognition and the text-to-speech technology is universal at present, most commercial system cannot provide an accurate and intuitive interaction experience. In such background, some researchers hope the study of relations between the articulatory and acoustic could inspire the speech signal processing research. Nowadays, many sensors has been developed to build these relationships.In this thesis, a framework which synthesize the continuous ultrasound based articulatory movement from speech signal is proposed. The system take the speech features as input and use mapping methods to synthesize the features of ultrasound images. After that, ultrasound images will be reconstructed from image features. At first, we describe the existing mapping methods: phonetic based and frame-by-frame based; next, the speech production model and Silent Speech Interface are introduced to give readers a better understanding of the application scenarios; then we present the detail of our multimodal ultrasound corpus;at last, the theory and implementation of k-means and Gaussian Mixture Model mapping methods are illustrated.The objective quantitative and subjective perceptive evaluations suggest our method is highly accurate and applicable. Our method could help understand relationships between articulatory and acoustic; visualize the movement of articulatory which can help the foreign language study, speech production rehabilitation and disease diagnosis; simulate and expand ultrasound based corpus.