Physiology Based Tongue Modeling And Speech Synchronized Animation Synthesis

For many years, researchers hope to simulate and synthesize articulators movements according to the kinetic mechanisms and principles of the articulators, for the use in the field of linguistics, clinical medicine, human-computer interaction, etc. Human articulators consist of tongue, lips, teetch, jaw, palate and so on. Among them, tongue is the most important, sophisticated and complex organ, and the position and movements of the tongue largely determine the pronunciation during speech, and thus research on the tongue modeling and simulation is very important. In the past, due to the lack of sufficient tongue data, it is difficult to simulate speech related tongue movements from the physiology perspective. Therefore, building a physiological accurate tongue model, and synthesizing speech synchronized tongue animation become a high challengeing task.In this paper, for the purpose of reflecting lifelike tongue physiological characteristics and synthesizing realistic tongue animation, we build a physiology based tongue model and speech synchronized animation system.The following will describe these specific research processes and innovations.1. This paper investigate tongue related anatomy and physiology, and proposes a tomography data and medical study based anatomical modeling method, and build a tongue mesh model with accurate geometry and intrinsic muscle architecture.First, the 3D tongue surface mesh is built according to the tongue tomography data, and then the tongue tetrahedral mesh with dense and regular tetrahedron is constructed by using meshing technology. Second, the tongue geometry and muscle fiber structure are constructed by a proposed interactive muscle marking method, and by using this marking method, tongue muscles are represented by the set of tetrahedron those distributed in the space formed by the muscles, and the muscle fibers are represented by the vectors on the nodes inside the muscles. Simulation results show our method is capable of reflecting precise tongue geometry and intrinsic muscle fiber structure.2. This paper researchs the kinetic mechanisms of the tongue, and proposes a finite element method based biomechanical modeling method, which is capable of reflecting accurate tongue tissue elastic properties and muscle movement characteristics.. By using this method, we build a physiology accurate tongue model which can be driven by muscle activations and can be used for tongue animation synthesis.First, the whole tongue model is endowed with a non-linear, quasi incompressible, isotropic, hyperelastic material properties, and for those tetrahedron marked as muscles, they are endowed with additional material properties, which reflect the muscle active and passive contraction properties, In this way, the movement of the tongue model can be driven by muscle activations through finite element analysis. Simulation results show our biomechanical modeling method can make the physiology tongue model reflect accurate tongue soft tissue elasticity and muscle physiological perpoties, and thus produce realisti c tongue aniamtion.For reflecting the dynamic effects of tongue movements, this paper presents a flexible constitutive modeling method, which fully takes into account the muscle speed-tension relationship, and this relationship is embedded into the constitutive modeling process of tongue muscle, and with this method, the deformation of the tongue model can be simulated either by a quasi-static way or by a dynamic way.3. For the purpose of certain applications, based on above physiology tongue model, this paper develops a speech synchronization tongue animation synthesis system, which can produce realistic tongue animation according to the 2D tongue movement data.First, lots of tongue deformation samples are produced according to lots of designed muscle activations, and these input and output samples are used to train a neural network that can transform muscle activations to tongue contours deformation directly. Second, the neural network is used to estimate corresponding movement parameters from collected tongue movement data, and the estimation results are used to build the physeme (sequences of muscle activations and rigid movement) database. Last, the phoneme sequences are transformed to the physemes sequences, which are then directly imputed to the physiology tongue model for simulation. Simulation results demonstrate that the synthesized tongue animations are visually realistic and approximate well with the collected tongue movement data.