Image-based Capture and Modeling of Dynamic Human Motion and Appearance

Photo-realistic renderings of humans are required for real-time graphics applications, and accurate human models are useful in applications such as model-based tracking. Non-rigid deformations of humans, e.g., deforming cloth and muscle bulging, are hard to model geometrically and are inefficient to simulate. Such deformations are often dependent on the subjects kinematic pose. In this thesis, an image-based pipeline is used to acquire a compact model of these non-rigid deformations. The model is capable of generating photo-realistic renderings of deformation and appearance effects from novel animation and viewpoint.;The compact model of non-rigid deformations is distilled from a multi-view training sequence exhibiting the desired pose-dependent deformations. The geometric deformations are modeled with a pose-dependent geometry attached to a kinematic skeleton, and the appearance is modeled with a pose- and view-dependent appearance basis. Pose-dependent effects not encoded in the geometry are represented in the appearance, and view-dependent appearance compensates for inaccurate geometries and specular effects.;In acquisition, a base geometry is recovered from a static multi-view image sequence. For human geometries, a two camera turntable-based acquisition is proposed. The acquisition interleaves tracking and silhouette refinement to account for unintentional motion of the subject.;The coarse motion of the base geometry is tracked in a separate training sequence using a common tracking formulation for linear blend skinned meshes. The energy formulation combines silhouette or intensity-based data terms with pose prior and smoothness terms. Local optimization with GPU acceleration gives near-real time results on intensity-based terms.;The recovery of fine-scale geometric deformations necessary to build the model is studied in situations with a few or non-overlapping views. For a moving monocular camera, a simple constant velocity constraint is shown to enable the reconstruction of both dense scene flow and structure in a variational formulation. This simple constraint is generalized to a multi-view setting, where long range flow is represented with a temporal motion basis layered on top of a geometric proxy surface.;The complete compact model is demonstrated on several examples, including modeling of cloth deformation on arms, transferring the appearance of wrinkles on pants to novel walk cycles, and applications of free-viewpoint compression.