View Synthesis From Multi-view Video

3D TV is a novel medium developed in recent years that is able to enhance video viewing experience. Based on multi-view video technology, which can synthesis a range of arbitrary view by depth-image-based rendering (DIBR), a 3D TV system can provide the viewer the interactivity and stereoscopic perception.The purpose of this dissertation is to probe into view synthesis from multi-view video based on DIBR, which will provide the theories and technical-support for its applications to 3D TV.In this dissertation, pin-hole camera model, the theoretical basis of view synthesis from multi-view video, is firstly introduced. Based on this model, two main aspects are studied: depth image acquisition and the key technologies of view synthesis based on DIBR. As for depth image acquisition, the epipolar line rectification for stereo pairs is thoroughly studied. As for view synthesis, a new hole filling algorithm is firstly proposed; then view synthesis based on image re-projection is discussed, and the method for arbitrary view synthesis based on DIBR is also presented; finianlly, de-interlacing of depth-image-based 3D video is elaborated.The main research results of this dissertation are summarized as follows.①With respect to epipolar line rectification for depth image acquisition, the theory of image re-projection is discussed; and based on image re-projection, a new method for epipolar line rectification for stereo pairs is given. The method combines the rectification method which needs the fundamental matrix with the rectification without the fundamental matrix, avoiding in this way local minima as well as the dependence on the precision of the fundamental matrix. Experimental results show that the presented method works accurately with low image distortion.②A new hole-filling algorithm is proposed. The algorithm guarantees that the holes which satisfy the following condition will be filled first: the number of non-holes in 4-neighborhood-pixels set is larger than or equal to the threshold of the current iteration. After each iteration, the algorithm will re-search the holes whose 4-neighborhood-pixels are all non-holes for the next iteration. Experiment results show that the algorithm can be employed efficiently for the small holes, especially for the holes at lowly-textured area. The results suggest that the algorithm adapts to the hole filling after depth image smoothing during the view synthesis procedure. In addition, the algorithm is facilitative for hardware implementation.③As for the special case when the camera rotates around its optical center, the properties of the homography in this case is thoroughly discusses. Combining the properties of the homography with the 3D image warping technique, the paper introduces a new way to synthesize novel views when camera rotates around its optical center is proposed. Due to the use of the properties of the homography, the proposed method can generate novel views with better image quality as well as faster rendering speed.④A new method of view synthesis based on DIBR is proposed. The method performs 3D image warping two times: the first warping uses the smoothed depth image, and the destination image contains holes with small hole-size; the second warping uses the original depth image, and usually big holes appear in the destination image. After that, two destination images are merged based on the imaged formed from the second warping. Finally, hole-filling algorithm is applied to the merged image. This method can avoid degrading the image quality in non-hole area when filling the holes, thus the"authenticity"of the image is guaranteed. Experiment results suggest that the method adapts to the stereo pair synthesis for 3DTV.⑤The transmition of the depth-image-based 3D video in current TV system is discussed. The conception of interlaced 3D video is proposed. Based on this concept, a de-interlacing algorithm for depth-image-based 3D video is given. Experiment results show that, as an intra-field interpolation algorithm based on direction is applied, the proposed de-interlacing algorithm can provide progressive novel views with quite good image quality, which also suggests that the method adapts to the de-interlacing of depth-image-based 3D video.As an innovative work, this thesis tried it's best to break through on research methods and research thoughts, and the research results will provide new methods for the practical applications of view synthesis from multi-view video in 3D TV.