Applications Of Speech Interface In Mandarin Learning Edutainment System

With Chinese economy development, the communication between China and the world becomes more and more frequent in a wide range. Mandarin, the communication tool and culture carrier that lets the foreign country know China. Enthusiasm for Mandarin learning has surged overseas in recent years. However, it is still facing many problems, such as teachers can't meet overseas students' demands and the traditional Mandarin teaching methods are not attractive or engaging for students. The edutainment learning styles can solve these problems well. Since the concept of serious game was proposed in the United States in 2004, it has been fruitful in many fields. However, today few people work on Mandarin learning edutainment development and there are no this kind of edutainment products existed.Aiming to solve the problem of Mandarin learning or teaching for foreign students, we designed an Edutainment system of Mandarin learning based on speech recognition, pronunciation evaluation, virtual Environment and serious game. Our research is show as follows:(1) We set up a Speaker-Independent Isolated Word Recognition System, using MFCC and pitch as the feature parameter, taking initial/final as the recognition unit, adopting HMM with 32 Gaussian mixture components per state as the acoustic model. Finally, we get a system with the WER lower than 2.0% which basically meets the need of reliable interaction and pronunciation evaluation in practice.(2) We improve the HTK recognizer HVITE running in the background, preparing for dealing with the unknown speech. The improved new recognizer can output two levels of information, word and initial/final.(3) We propose a new algorithm of pronunciation evaluation, which combines the log-likelihood based on HMM and segment duration on initial/final level. The method we proposed has higher correlation with human scoring than the log-likelihood based on HMM algorithm. Two unified mapping model of initial/final log-likelihood score have been established solving non-linear regression equations. And the speech interface of the Edutainment system is completed by embedding the initial/final pronunciation evaluation model into the HTK recognizer HVITE.(4) The system of Edutainment has been founded using Virtools. It is including virtual enviroment and some character models with animations. Some figure models in this system come from the Three-dimensional reconstruction fruits in our laboratory. We create a new speech recognition and pronunciation evaluation Building Blocks (BBs) as the speech interface using SDK of Virtools on VC++ 6.0 platform. It could provide the pronunciation information on initial/final level, helping students master their detail pronunciation information.