Data-driven Researches On 3D High-fidelity Hand Motion Modeling, Generation And Control

One of fundamental problems in computer graphics is the acquisition and gener-ation of realistic animation of human hand, and they are also important to many other areas such as film industry, video games, virtual reality, robotics, and human computer interaction (HCI). However, acquiring high-fidelity 3D hand motion data is a challeng-ing task, and there are no well proven approaches currently. Secondly, the generation of hand animation, especially the generation of realistic grasping animation of human hand, remains to be an open problem. Besides, the dynamic model of hand, such as skinned mesh model, is a necessity for the animation works of human hand. But the authoring of such models are labor sensitive and time consuming. This thesis does re-searches on some key problems of human hand animation from the data-centric view, to try to solve the questions above.Firstly, we proposed an effective framework for hand motion capture, by combin-ing the marker-based optical motion capture device, and the Kinect camera, to capture the high-fidelity 3D hand motion. Marker-based mocap has the advantage of high pre-cision and high time-resolution, but the data is very sparse, and markers may be miss-ing. Kinect has the advantage of high space-resolution, but the data precision is low. By combing the two which are complementary, we can achieve both advantages while avoiding their disadvantages. We have demonstrated the power and effectiveness of this system, by capturing a large amount of motion of human hand, including bare hand data, and more complicated grasping, i.e., hand-object interaction.Using the data of hand motion capture, we have proposed a realtime synthesis and motion control algorithm for human grasping. We first segment the grasping process into three phases, namely reaching, closing, and manipulation. Then separate data-driven algorithms are built for each phase, to synthesize the corresponding motion. The results have the realistic appearance, and also show the diversity. We then introduced the physics-based motion control algorithm, to compute the joint torques needed for hand to grasp the object, which are then used to drive the hand for physics simula-tion. The final motion is physically realistic, and can react to external perturbations and changes in physical quantities. Besides, we also developed a performance interface which allows users to act out the desired grasping motion in front of a Kinect camera. We demonstrated the power of our system by generating realistic motions for grasp- ing objects with different properties such as shapes, weights, spatial orientations, and frictions.To create the user-specific hand skin model to serve as the basis of hand motion acquisition and generation, we have proposed a data-driven approach to model the pose and shape for human hand. Firstly we build a database by scanning a large set of 3D static meshes of different subjects under different poses. Then our models are learned from this database. We use linear blend skinning (LBS) for pose model, as it is de-facto standard and is compatible with the existing animation systems. The pose model parameter is the very skinning weight map. The shape model is decomposed into two components:skeleton size and vertex displacement, and we also get the compact rep-resentations for them using principle component analysis. Using the models learned, we can reconstruct the user-specific hand skin model from different types of input, such as RGB images, Kinect camera, incomplete scan mesh, and even semantic data. This greatly reduces the works needed in traditional way. We also demonstrated other inter-esting applications, such as skin transfer, and model-based 3D hand pose tracking.Apart from technical contributions, the works of this thesis have also led to some data sets for human hand, such as motion data for bare hand, motion data for object grasping, and hand model data. Besides, we have also developed some practical tools and systems, for example, high fidelity hand motion capture system, realtime anima-tion synthesis system for human grasping, user-specific hand skin model reconstruction system, and skin transfer system. The data as well as these systems will advance the researches in the animation filed of human hand.