Kinect Based Active Appearance Models And Their Application On Expression Animation

Locating and tracking facial features, which is widely used in face recognition, gesture, expression analysis, expression transfer, human-computer interaction and psychological cognition, is a very active research direction in computer vision. It also provides input for many applications, such as identification, online chat, video compression, special film effects, human-computer interaction, etc. However, because of the complexity of the facial expression, locating and tracking facial features is still a hot spot in the research of this field. The facial features are mainly around the eyes, mouth, eyebrows, and outer contour, such as the corners of the mouth. The position of feature points can be obtained through image analysis.Locating and tracking facial features in video under conditions of complex backgrounds, illumination changes and pose changes is difficult if only relying on RGB image. In recent years, more and more studies tend to use the3D model and depth information. Kinect can provide depth information and RGB image information at the same time, which provides a new tool for facial feature extraction in the video.After extracting facial features from video, we can analyze the expression information of each frame and transfer the expression to the target face. Because of the difference of the individual faces and the expressions, suitable mapping relationship from features information to the target face expressions is need to be find.Therefore, for the work of facial expressions transfer from the video, two problems need to be solved in this paper. The first is how to extract expressions or feature points from video information; the second is how to transfer to the target animation after extraction. This article will revolve around these two aspects, which includes the followings.(1) Locating and Tracking the facial features from video frames that contain both depth information and the2D RGB image. This aspect of the study contains two parts, one part is the multiview, namely the head rotation cases, facial feature points location problem, the other part is the features location problem under the restriction of the facial contour.1) Multiview Active Appearance Model Based on KinectThe precision and robustness is the key to the study of facial feature points extraction. The extraction of facial feature points is susceptible to illumination and pose that bring a lot of matching error. However, due to the use of infrared,3D depth data extracted from Kinect is less affected by illumination and the introduced depth information could enhance the accuracy of the pose decision. Due to the additional depth data and2D image are from dual cameras, angle difference exists between the cameras, the pixels are not a one-to-one correspondence that need to be solved to obtain the depth of each pixel value. At the same time, the quality of the initialization has much effect on the active appearance model. Thus, how to initialize the matching model under the multiview conditions after head pose extracting is also one of our research questions.2) Coutour Constraints based Facial Coutour Features LocatingTraditional active appearance model use image information to match features, but in complex background environment, it is difficult to initialize the human face location and easily influenced by background. With depth camera, from the depth data, it is easy to find the edge of the face, which can be used as constraints to locate the facial features on the edge, thus improve the matching accuracy. How to constraint the facial features after introducing three-dimensional information is also one of our research questions in this paper.(2) Video Driven Facial Line Drawing AnimationFacial line drawing animation is a full of vitality art form, the successive animation production from video sequences need to study how to extract expression from video as simple as possible, extract the facial expression characteristic coefficients, and transfer it to the line drawing animation character.The main contents and contributions of this paper are as follows:(1) To solve the initialization and head motion problem of active appearance models, We propose a convenient method for initialization and model choosing of multiview AAM under depth data from video sequence, which enhance the accuracy of the algorithm.We propose a method, which use the depth and RGB image, to extend the active appearance models. Based on the view based AAM models which are trained by sampling different perspective images, we propose a facial3D grid model establish method based on facial feature points and depth data, then initialize the AAM and choose appropriate model at different views through this grid, that improved the accuracy and robustness of AAM.(2) To solve the accuracy problem of contour features, which caused by that only one side texture at the contour take part in the fit optimization process, we proposed a method that can constrain the contour features with the depth data, that can effectively improve the coutour feature points fitting accuracy.The idea of the fitting process is as follows. Firstly, extract the head area from depth data with threshold and cut out the face, then extract the coutour of face. Secondly, our method initializes AAM and iterates the fitting process to find feature points, which are close to the result, then find the nearest points of every feature points on coutour of face. After that we add those points as constrains to the followed iterations which makes the result is more close to the real coutour. Experiment shows that algorithm can improve the accuracy of coutour features. However the accuracy of inner features is a little reduced that would be caused by the global constrain process such that the inner features would be dragged from the original position to satisfy the contour features.(3) To solve the problem of transferring expression from video to the line drawing animation, we proposed a method that uses the shape coefficients of AAM fitting process to analyze the expression and transfer it to line drawing animation. We design an application tool to realize that.The line drawing model use a fixed background and use the Bezier curves with width to express eyes, eyebrows, and lip. We extract the current position of the facial feature points from input video by AAM, and then decompose those into combination of expressions of a particular user’s basis emotions. At last, we drive the line drawing model with the combination coefficients to create animation. This method is easy to implement and need few basic emotions that is convenient for transport and storage. The output can keep the expression changes of origin video and can be further edited with the provided tools. Because of the use of coefficients combination method, this method can be easily extended to other character model.