Recovery of three dimensional scene flow and structure

This dissertation addresses the problem of automatically recovering three-dimensional structure and scene flow of a common scene. We develop different algorithms under different camera setups to improve the robustness and accuracy of structure and motion estimates.; For the monocular view case, we design a novel framework that is able to automatically track 3D rigid objects under partial occlusion in monocular image sequences. We show that motion residual errors can be used to detect the occluded areas during tracking. We implement a 3D human head tracking system and carry out extensive experiments on both synthetic and real imagery. Promising results have been reported.; For the multiview case, we explore the problem of stereo matching and scene flow computation. We design three novel stereo matching algorithms: an EOFC-based algorithm utilizes the modified extended optical flow constraint to solve the stereo matching problem; a rule-based stereo matching algorithm reduces the fattening effects that are common in most stereo matching methods; a segmentation-based cooperative algorithm has all the advantages of the segmentation-based methods (i.e., discontinuities are well preserved), while it estimates disparity values in a more global manner. Extensive experiments are carried out to compare among the proposed algorithms and other state-of-the-art stereo matching algorithms, both quantitatively and qualitatively, and the effectiveness of our algorithms is demonstrated. We also investigate the problem of recovering the dense 3D non-rigid scene flow from multiview image sequences. We develop two systems computing dense non-rigid 3D scene flow and structure from multiview image sequences. The first system, Integrated Model-based System (IMS), integrates structure and motion in a mutually beneficial way. The second system, Extended Gradient-based System (EGS), can be thought of as a natural extension of 2D optical flow computation. This system uses image segmentation information to adopt and maintain the motion and depth discontinuities. Experimental results on both synthetic and real imagery demonstrate the effectiveness of our 3D motion and structure recovery schemes. Empirical Comparison between IMS and EGS is also reported.