Realistic Mouth Image Synthesis

Face image synthesis is playing an increasing important role in multi-modal human-computer interfaces. The potential applications of this technique include low-band video transmission, joint audio-video coding, film making, computer game and etc. However, generating realistic face animation is a hard work. In order to make realistic sense, not only all the details in the face should be rendered as naturally as possible, but also dynamic texture and non-permanent features should be rendered vividly. The mouth images are difficult to synthesize due to large variation. The conventional approaches such as manipulating wire-frame model or warping key-frames cannot render all dynamic features of mouth. To solve the puzzle, the paper proposes a novel solution, which includes building parametric model of mouth and developing a dependence mapping from shape to gray-level texture. Thus, realistic mouth image can be synthesized completely by a few shape parameters.There are many reasons to cause the mouth images variable, such as different individuals, illumination, mouth opening, especially visibility of teeth and tongue. For the convenience of analysis and synthesis, it should build parametric model of mouth images. Referencing Cootes et al's flexible model, we separate overall appearance of mouth into shape part and shape-free gray-level part, and build the shape model and gray-level model by performing statistic analysis over a training set of mouth images. Since the models are derived by principle component analysis, they can represent most variation of mouth image. After building the flexible model, given a mouth image, we can calculate its corresponding model parameters. In reverse, given a set of model parameters, we can synthesize overall appearance of the mouth.The shape and shape-free gray-level are two complement parts describing overall appearance of mouth. But it is obvious that the mouth shape and gray-level texture inside have some dependence, we represent the dependence as a mapping, and develop the mapping using training set. Therefore, overall appearance of mouth can be synthesized from a few shape parameters. Although there are some factors in nature and difficulties in implementation make the relationship between shape and gray-level to form a mapping, the experimental results are exciting. With the non-linear mapping implemented by feed-forward neural network, we can successfully estimate gray-level parameters by shape parameters, the final synthesis results are very close to real images.