A Study Of The Mapping Between Articulations And Acoustics

Constructing a mapping between articulatory movements and corresponding speech could significantly facilitate speech training and the development of speech aids for voice disorder patients.In this paper,we record the audio when pronouncing the Chinese vowels(a,o,e,i,u,ü)with audio recorder,the movements of my tongue with ultrasound and the front and side motion of lips with high-speed camera,thus forming a multimodal Chinese vowels speech database.We propose a novel deep learning framework for the creation of a bidirectional mapping between articulatory information and synchronized speech recorded using an ultrasound system.We created a dataset comprising six Chinese vowels and employed the Bimodal Deep Autoencoders algorithm based on the Restricted Boltzmann Machine(RBM)to extract the feature of articulatory information and acoustics.Speech and ultrasound images were then reconstructed from the extracted features.The reconstruction error of the ultrasound images created with our method was found to be less than that of the approach based on Principal Components Analysis(PCA).Further,the reconstructed speech approximated the original as the mean formants error(MFE)was small.Then,we carried out mapping between ultrasound images of the tongue and corresponding acoustics signals with a Deep Neural Network(DNN)framework using the revised Deep Denoising Autoencoders.The results obtained indicate that the performance of our proposed method is better than that of a Gaussian Mixture Model(GMM)-based method to which it was compared.Also,we apply this framework to build the mapping based on continuous speech database,and the performance is promising.