A self-correcting stereo vision system for view synthesis

The view synthesis process constructs realistic images of a scene, as if taken from new viewpoints, using correspondence information between reference images of the same scene. While early view synthesis methods employed manually-generated correspondence data, recent methods have begun to use stereo computer vision to gather correspondence data. The results, although promising, demand improvement in both the automatic correspondence methods and the synthesis algorithms.; The correspondence methods for view synthesis suffer from many of the traditional problems of computer stereo. First, the correspondence search range is often either too large or too small. Usually set during design time, the search range does not tolerate a dynamic scene. Second, although there can be no correspondence in half-occluded regions, many stereo systems do not detect these regions and provide spurious matching results therein. Third, current methods to predict correspondence mismatch in stereo are inadequate, providing many false positives. Other problems that beset traditional stereo also affect view synthesis stereo systems. These problems include low-texture patches, sensor noise in the system, and lighting change between the reference images.; Even with a perfect correspondence map, many synthesis methods still produce distorted results. The most notable artifact comes from half-occluded regions. Since there is no correspondence in these regions, one must estimate a disparity that produces believable results in the synthesized images. When multiple bodies create half-occlusions, current synthesis methods fail.; In the dissertation, we investigate improvements to view synthesis that make the system more ‘introspective’ in the sense that the system detects and reacts to its own errors. For the correspondence problem, we propose a new method to integrate silhouettes and stereo matching to produce a better reconstruction. Also, we propose a new method to estimate stereo error. We develop new methods to detect half-occlusions and compare these new methods with current techniques. For the synthesis problem, we describe a novel view synthesis algorithm that copes with half-occlusion and produces believable results. Throughout these experiments, we provide both qualitative and quantitative data to validate the new approaches. We demonstrate an automatic view synthesis algorithm that can produce believable results for dynamic scenes.