| We present a novel approach to joint space-time, motion-based video segmentation and occlusion detection. The segmentation of an image sequence into moving objects and estimation of object motion belong to the most important tasks in image sequence analysis. Image sequence segmentation is a very difficult problem, with numerous applications, including content-dependent video compression (e.g., MPEG-4), video processing (e.g., object-based frame-rate conversion and deinterlacing), surveillance, video database queries (event detection, tracking), and computer vision (scene analysis, structure from motion). In most studies to date, image sequences have been primarily analyzed and processed in groups of two frames; by differentiating one frame from the other, one is able to infer the dynamics occurring in an image sequence. These short-term dynamics (such as displacement between two frames, or occlusion/exposure areas) can be linked together or temporally constrained in order to reason about longer term dynamics. Although the two-frame approach has been very successful in some applications (e.g., MPEG compression standards), it is often inadequate for the analysis of non-constant velocity motion, detection of long-term innovation areas (occlusion and exposure), or video segmentation.; In this dissertation, we propose to perform image sequence analysis jointly over multiple frames. We concentrate on spatio-temporal segmentation of image sequences and on the analysis of occlusion effects therein. The segmentation process is three-dimensional (3-D); we search for a volume carved out by each moving object in the image sequence domain, or "object tunnel", a new space-time concept. We pose the problem in variational framework by using only motion information (no intensity edges). The resulting problem can be viewed as volume competition, a 3-D generalization of region competition. We parameterize the unknown surface to be estimated, but rather than solving for it using an active-surface approach, we embed it into a higher-dimensional function and apply level-set methodology. We first develop simple models for the detection of moving objects over static background; no motion models are needed. Then, in order to improve segmentation accuracy, we incorporate parametric motion models (affine) for objects and background. We further extend the method by including explicit models for occluded and newly-exposed areas that lead to "occlusion volumes", another new spacetime concept. Since in this case multiple volumes are sought, we apply a multiphase version of the level-set method. We extend our motion detection to account for camera motion and zoom-in (background is no longer static). In order to reduce computational complexity of our methods, we apply a recently-proposed fast level-set implementation and investigate its performance. We present various experimental results for synthetic and natural image sequences, including those from the VIVID Tracking Evaluation Web Site at Carnegie Mellon University. |
