Motion segmentation and dense reconstruction of scenes containing moving objects observed by a moving camera

We investigate two fundamental issues in Computer Vision: 2D motion segmentation and 3D dense shape reconstruction of a dynamic scene observed from a moving camera. The scene contains multiple rigid objects moving in a static background, while the camera undergoes general 3D rotation and translation. Our goal is to segment the video frames into 2D motion regions and static background areas, and then to reconstruct the dense 3D shape of both parts of the scene.;Motion segmentation of image sequences shot by a moving camera is inherently difficult as the camera motion induces a displacement for all the image pixels. This camera motion is compensated for by a number of geometric constraints estimated between video frames. The pixels that cannot be compensated for by these constraints are classified as motion regions. A novel 3-view constraint is proposed to handle the cases where existing ones do not work well. The geometric constraints are combined in a decision tree based method for segmenting the motion regions from the background area in each video frame.;After motion segmentation, sparse 3D structure of the static background and 3D camera motion are estimated by the well-developed "Structure and Motion (SaM)" methods. The same SaM methods are applied to recover the 3D shape of moving objects from a moving camera, based on their relative motion. The object scale and motion, however, can be only solved up to an unknown scale, unless additional assumptions are available. In our scenario, a planar-motion assumption is introduced: the object motion trajectory must be parallel to a plane. With the aid of the planar-motion assumption, the 3D object motion trajectory can be uniquely determined.;The sparse 3D reconstruction of the dynamic scene is extended to a dense volumetric one. The whole scene is divided into a set of volume elements, termed as voxels. Each voxel is assigned an object label which may change over time. The task of dense reconstruction is then accomplished by a novel voxel coloring method that finds the optimal label assignment for each voxel to minimize photo-motion variance measures between the voxels and the original images.