| This dissertation concentrates on the problem of intermediate view reconstruction, which is defined as follows: given few images of a scene captured by real cameras, reconstruct images that would have been captured by virtual cameras. Our main goal in this dissertation is to reconstruct intermediate views using two input images with a special focus on occlusion areas. Occlusion areas are the areas that are visible only in one of the input images.; We start the dissertation by identifying the main challenges in intermediate view reconstruction from two images, and then offer novel solutions to each challenge. First, we present in detail the popular pivoting-based view reconstruction that requires estimation of a separate disparity field for each view under reconstruction. After pointing out the deficiencies of pivoting-based approach, as an alternative, we propose a new intermediate view reconstruction method based on B-spline approximation. The new approach permits reconstruction of multiple views from a single disparity field, a clear computational advantage. It also assures better robustness to image noise, although is more sensitive to disparity estimation errors than the pivoting-based method. However, most importantly, spline-based reconstruction allows selective forward compensation of visible areas and, therefore, is of importance in occlusion awareness. Next, we present a new simple occlusion area estimation method and show its superior performance over other low-complexity algorithms. The knowledge where occlusions occur in an image is a valuable piece of information since disparity cannot be reliably estimated there and needs to be inferred in a different manner. In view of this, we present a novel approach to disparity recovery in occlusion areas, namely the image-driven disparity inpainting. We further embed this idea into a variational formulation, and propose occlusion-aware optical flow (disparity) estimation that jointly computes disparity vectors, implicitly detects occlusions and extrapolates disparities in occlusion areas. Combining all of these proposed methods in view reconstruction, we reconstruct realistic and improved intermediate views especially in occlusion areas. Finally, we focus on using multiple images, instead of two images, in view reconstruction to improve the pivoting-based approach. Specifically, we propose another occlusion-aware pivoting-based disparity estimation formulation, which adaptively estimates disparity by using different pairs of input images. The reconstruction using multiple images shows significant improvements over pivoting-based reconstruction that uses two images only.; Intermediate view reconstruction has many applications, especially in the area of 3D displays and communication. One of the most important applications is that it can be used to reconstruct additional views from stereo pairs, so that any stereo pair can be displayed on emerging automultiscopic displays that require many views of a scene as input. It is in this context that we demonstrate applications of the proposed methods in different areas. First, in a biomedical application, the proposed view reconstruction algorithm is embedded into a neuromuscular training system that utilizes automultiscopic 3D displays. Second, the proposed view reconstruction method is applied to monoscopic video sequences to increase frame rate and, therefore, enhance low frame-rate videos captured by mobile phones. Third, the proposed occlusion-aware optical flow method is used to solve a real-world problem of NASA, namely the recovery of a missing color component in stereo images. Finally, disparity estimation in a multiview video codec that uses view reconstruction is shown to benefit from our methods. |
