Research On The Up/Downsampling Methods Of Depth Image For 3D Scene

With the popularization of 3D movies,more and more people begin to get in more touch with different 3D applications,promoting the continuously progress of 3D technologies.Many 3D applications such as driverless cars,virtual reality are also developed rapidly.Compared with 2D applications,3D applications give users more real stereo experience by introducing depth information and reconstructing the whole 3D scene with color information.Depth information reflects the distance between the objects in the scene and the camera.It is usually encoded as a gray image,names as depth image.Compared with color images,depth images are smoother at most regions and sparser.Furthermore,depth images are used for synthesizing virtual views with color images instead of for direct watching.Edges in depth images reflect the latent discontinuities of the real scene,therefore,it is more important to recover accurate and sharp edges in depth images than in color images.However,due to the limitation of physical equipment and the acquiring principle,the resolutions of the depth images captured are quite limited.In this situation,it is quite essential to increase the resolutions of depth images by using post processing.Moreover,the introducing of depth images improves the 3D experience but also increases the quantity of data.In a band limited situation,it is an important problem to recover better depth information by choosing the transferred depth samples wisely.In order to solve such problems,this paper focuses on the following three aspects.The first work of this dissertation focuses on single depth image upsampling.Due to the limitation of the acquiring equipment and the principle,the resolutions of the depth images captured from the depth acquisition equipment are limited and improper for practical applications.Studies indicate that blurry edges in depth image result in jagging artifacts in the synthesized views.In order to solve these problems,the property of depth image is analyzed,and a local planar approximation for depth images is proposed.After that,a single depth image upsampling algorithm based on local planar approximation is further proposed.In this algorithm,finite candidate depth values are first generated using the acquired depth values nearby according to the local planar approximation.Then a guidance gradient field is obtained by using gradient profile prior,to constrain the combination of those candidate depth values.Finally,the high resolution depth image is derived by solve the constrained optimization problem.Experimental results verify the rationality of the local planar approximation and demonstrate that the proposed algorithm recovers high resolution depth image with sharper edges.In the second part of this dissertation,we focus on the color guided depth image upsampling.By introducing an auxiliary high resolution color image,color guided depth image upsampling algorithms improve the quality of the upsampled depth image by exploiting the property that color edges and depth discontinuities usually appear together at the boundaries of objects.However,the inconsistence of depth discontinuities and color edges causes texture copying artifacts and edge blurring artifacts.In this paper,we propose a spatially adaptive tensor total variation-Tikhonov regularization model for depth image upsampling while avoiding abovementioned artifacts.Considering the importance of edges in depth images,we propose to use guided tensor total variation regularization for edge preserving upsampling.Mathematical analysis of the guided tensor total variation proves that this regularization preserves and aligns edges but introduces staircasing artifacts,therefore a Tikhonov regularization term is imposed to suppress these artifacts in smooth regions in depth images.In order to avoid texture copying artifacts and edge blurring artifacts introduced by the inconsistence of depth discontinuities and color edges,a fused edge map is proposed to guide tensor total variation.The fused edge map utilizes the different appearance of color edges and depth edges to indicate discontinuities regions in depth images.Finally,the tensor total variation regularization term and the Tikhonov regularization term are spatially adaptive combined to preserve edges and recover smooth results in different regions.Specially,a first order primal dual algorithm is adopted to solve this convex but not differentiable model.Experimental results show that the proposed method recover high resolution depth image effectively.Moreover,the proposed method yields much sharper edges and lower percentage of bad pixels.At the last part of this dissertation,we analyze the effective sampling strategy at a given sampling ratio.The introducing of depth information increases the total amount number of data to be transferred,however,the bandwidth is usually limited in applications.It is a challenge problem to choose proper depth samples to be transferred in a fixed bandwidth environment,to recover better depth image in the receiving terminal.In order to solve this problem,a nonuniform depth downsampling method utilizing both the properties of depth image and human visual system is proposed.Since edges is the most important feature in depth image,samples in edge regions should be well preserved firstly.Moreover,human pays more attention to salient regions than background regions thus more samples should be assigned to salient regions.Combined the above two properties,we proposed to generate a fused saliency map by fusing edge information of depth image and salience regions,to indicate the different importance of regions in depth images and then use a multi-level sampling strategy to generate samples wisely.Experimental results demonstrate that the proposed method recovers better high resolution depth images in the same sampling ratio,compared with uniform random sampling and uniform grid sampling schemes.The dissertation researches the up/downsampling methods for depth image,the research on depth image upsampling can be further used in depth image enhancement to improve the quality of 3D applications;the research on depth image downsampling can be applied for effective depth acquisition and depth image compression.In general,the researches in this dissertation can be applied in various 3D applications based on depth images,providing new ways and solutions to improve the quality of different applications.