Layered deformotion with radiance: A model for appearance, segmentation, registration, and tracking

This dissertation gives a general model for the estimation of shape (image segmentation), appearance, pose (image registration), and movement (tracking). The model can infer parameters for multiple objects in a dynamically changing scene. This of course has a number of real-world applications. In particular, in visual tracking, moving the camera to keep objects of interest in the field of view may cause the background to move. The objects can move and deform in three dimensions, but they must be captured in two-dimensional images.; Each component of the image is represented by a separate layer: one for the background and a layer for each foreground object. Each layer has three components: a contour that bounds the region of the layer, a smooth function that represents the object's appearance, and a transformation that maps that layer into an image. The segmentation for each layer is a contour (embedded as the zero level set of a distance function) that is the average shape of the object computed from multiple images. The smooth function associated with a layer approximates the image data inside the contour, after the contour has been mapped into the image by a similarity transformation (rigid component) plus a vector field (non-rigid component). A very practical application of having this model is that one can fix the size of a layer and then construct priors on both shape and appearance for that layer. These priors are constructed using principal components analysis (PCA) which reduces the dimensionality of the image-approximating smooth function and the vector field (non-rigid registration) and allows for more accurate modeling of an object for that layer.