Hole-filling For Depth-image-based Rendering

With the rapid development of 3D display technology in recent years, 3D TV has become one of the important electronic products in our daily lives. Depth-image-based rendering(DIBR) is the key technique of 3D display. It can synthesize the arbitrary view within a certain range, which makes 3D TV system interactive and stereoscopic. However, this technique has a flaw: it may generate “holes” in the synthesized view.The propose of this dissertation is to propose high-performance hole-filling algorithms, improve the quality of the synthesized view, and provide the theoretical and technical support for the wide application of DIBR in 3D TV system.This dissertation mainly studied the hole-filling algorithms of view synthesis based on DIBR. Three hole-filling algorithms are proposed according to the characteristics of the holes: hole-filling based on MV General 3D image warping, hole-filling based on Spatio-temporal search, and hole-filling based on Delaunay triangulation.The main research results of this dissertation are summarized as follows:(1)In order to find the image contents that can be used to fill the holes in the destination image, the concept of the motion vector(MV) general 3D image warping is introduced, and the hole-filling algorithm based on MV general 3D image warping is proposed. The MV General 3D image warping functions for parallel camera setup and unparalleled camera setup is studied. The properties and correctness of these functions are verified. Two methods were used to fill holes: 1) MVs from reference image were utilized and the contents of other reference sequences were copied; 2) MVs of destination image were utilized and the contents from other destination sequences were copied. Experimental results show that the latter can achieve better performance. This hole-filling algorithm is suitable for filling the holes generated by moving objects.(2)In order to ensure the spatiotemporal consistency of the filled region, hence improve the image quality of synthesized views, hole-filling based on Spatio-temporal search is proposed. This algorithm uses the destination sequences and the reference image of the current destination image to fill holes. Firstly, a sequence-disoccluded region(SDR) search algorithm is introduced to find the real contents for hole-filling. Then, the remaining holes are filled using the contents of the reference image determined by the inverse of 3D image wrapping. Finally, image inpainting method is applied to fill the marginal holes. The experimental results show that this method not only greatly reduces artifacts as well as ensures the spatiotemporal consistency, but also avoid the image quality degrading in non-hole areas in contrast to pre-processing-of-depth-image methods.(3)As for the holes generated by rotating or scaling objects in the destination image, a hole-filling method based on Delaunay triangulation is proposed. This method makes the best use of the properties of Delaunay triangulation in image morphing technique. First, feature points detection and Delaunay triangulation are performed in the depth map. Then, the points in the triangle in the reference image corresponding to the hole points in the triangle of the destination image are determined by the triangle mapping. Next, a bilinear interpolation method is applied to determine the pixels for filling. Finally, the marginal holes are filled using the contents of the reference image. Experimental results show that this method is suitable for the scenario that contains rotating or scaling objects.The research results will provide new methods for the practical applications of DIBR-based view synthesis in 3D TV.