Research On Stereo Matching And View Synthesis For Natural3DTV System

Three-dimensional perception of the true world is one of human's intrinsic experiences. In recent years, with the advance and development of computer graphics, computer network and digital multimedia technologies, three-dimensional television will be the next-generation digital television service. Natural three-dimensional television expects to provide audience real world impression via natural scene capture, traditional three-dimensional television compatible transmission and autostereocopic display without glasses.Content presentation and view reconstruction are two key modules of natural three-dimensional television system. Content presentation refers to extract depth information of spatial scene location relationship from multi-view videos. And the depth information together with color image forms the basic data format of three-dimensional television. View reconstruction refers to the process of constructing multi-view scene from color and depth images. This paper analyses the basic requirements and goals of natural three-dimensional television, then researches and proposes suitable methods for content presentation and view reconstruction. The main work and contributions are summarized as follows:1. A sequential updating belief propagation (BP) stereo matching algorithm based on Markov random fields (MRF) is proposed. This algorithm can achieve fine matching result and decrease convergence time. With the aid of two-directional BP disparity estimation, the hole areas of interpolation view are effectively reduced in view reconstruction process.2. A scanline based optimization algorithm is proposed, which refines the basic steps of stereo matching. First, the algorithm adopts cross based cost aggregation strategy which greatly improves matching stability. In step of optimization, the algorithm contains two stages which are named optimization along scanlines and optimization across scanlines. The computation complexity is decreased while achieving good optimization results. At last, cross check is used to refine the disparity results on object boundary. Due to the parallel computation proficiency of GPU, the algorithm runs at about35fps on a typical image with resolution of384×288and16disparity levels. The objective accuracy results on middlebury website show that the algorithm outperforms other real-time\scanline-optimization matching algorithms. 3. In order to solve the temporal inconsistency problem when extracting disparity sequence from stereo videos using a frame-wise matching algorithm, two effective methods are proposed. The first one is spatio-temporal BP algorithm based on3D-MRF. As directly solving the problem is too complicated with expensive time and space complexity, a temporally streamlined framework is constructed. The result shows that the method can convincingly refine the temporal consistency of disparity maps without degradation of synthesized image quality.The second class of spatio-temporal stereo matching algorithm is based on optical flow. The algorithm uses BP on spatial domain, and preserves the belief values as frame-wise matching cost. Then in temporal domain, optical flow is estimated and used to model the smoothness cost function between two consecutive frames. The disparity is finally determined by dynamic programming. Also, the method is temporally streamlined, which ensures the disparity sequence consistency is improved while maintaining low computation cost.4. This paper solves the visibility, re-sampling and hole filling problems in the process of multi-view reconstruction through depth (disparity) image based virtual view rendering techniques. Especially, a MRF-based disocclusion restoration algorithm is proposed which combines the merits of structure inpainting and texture inpainting. Then the hole region information is estimated by global optimization. Meanwhile, this paper proposes methods of occlusion detection and presentation. Based on "video"+"depth"+"occlusion" data presentation, the synthesized view quality can be improved under a relatively low bandwidth.