Acoustic-to-Mouth Model Mapping System Based On Kinect 3D Data

Language is a human-specific feature and is the special abilities of human being different from other creatures.With the advent of world integration,the number of language learners is increasing at a very fast pace.In this case,computer aided pronunciation training(CAPT)technology emerges,which can not only ease the pressure on language teacher,but also provide round-the-clock service and support self-based training.This paper proposes an acoustic-to-mouth model system.In this system,firstly,using the audio and the 3D data recorded by Kinect as the source data,the mapping model is trained with mapping method,next,the 3D data is acquired through input the corresponding sound signal into the mapping model,lastly,a 3D model of the lip can be built according to the obtained 3D data.This paper has researched the k-means-based method and Gaussian mixture model-base method respectively.The mapping result were evaluated from the subjective and objective aspects,and the two mapping method were compared.In order to improve the mapping results,this paper also has used the low pass filter to smooth the mapping result.Experimental results show that the mapping method based on Gaussian mixture model is better than that based on k-means,and also confirms that the late use of low pass filter for smoothing can effectively improve test result.This system can provide audio and lip movement information for users,enabling users to better pronunciation training,so that it is expected to be used for pronunciation therapy and second language learning.