Research On Real-Time Speech Synchronized Expressions And Gestures Of 3D Avatar

Usually, when people communicating with each other, in addition to presenting the natural facial expressions, but also always accompanied by nonverbal facial gestures, the so-called facial gestures include head movements, eye movements and eyebrows movements, etc. These natural facial expressions and gestures play an auxiliary role to the speech, and also reflect the speaker’s inner emotional state. Currently, researches in this area are mostly limited to one side, when the avatar is speaking, few people consider the coordination and synchronization of the speech, facial expression, head and eye movements, etc. At the same time, the details of expressions and the problems of continuous transition are seldom considered in the simulation, so that the avatar looks lifeless and relatively simple, it has a great difference compared with the real human expressions. The goal of this thesis is to explore a new method for reproducing facial expressions which seem to be more believable and building a realistic emotional talking avatar system.Aiming at the problem of facial expression is relatively simple, no rich expression details, the expressions and the facial gestures can’t be well coordinated on the existing talking avatar, firstly, an automatic generation method of example-based blendshapes is proposed, the blendshapes as the basic model, we have used an new approach based on blend shape function to synthesis complex expressions which include the features of multi-expressions. In order to simulate the dynamic facial expressions, the changing process of the expression is divided into three stages:Onset, Apex, Offset, we have obtained good simulation results by this three temporal parameters method. Secondly, by studying the statistical data of eye and head movements from human psychology, a uniformly distribution random function which following some basic laws is used to simulate natural behaviors of blinking of eyes, gaze shifts and head movements. Thirdly, we have analyzed the relationship between dynamic expressions and spoken speech. In order for the TTS (Text To Speech) system to recognize expressions, we have converted expression tags into bookmark tags, and then bookmark function is used to achieve the synchronization between spoken speech and dynamic facial expressions.Based on the above research, a realistic emotional talking avatar system has been established in this thesis, which can show relatively real facial animation according to the input text information. The results of the experiment indicate the avatar looks not only realistic and full of emotion, but also the voice, dynamic facial expressions and facial gestures have reached a very good coordination and synchronization.