Research On High Quality Depth Maps Acquisition For RGB-D Data

With the development of computer vision,the problem of high-quality depth map acquisition is demanding urgent solutions.Generally,the approaches for depth map acquisition consist of two categories:passive and active methods.Passive methods based on stereo matching algorithm always compute matching cost volume pixel by pixel,which is time-consuming.This thesis firstly proposes a local depth estimation method based on adaptive matching scheme.Furthermore,with the help of affine invariant feature,the performance of matching in smooth regions is improved.Experimental results show that the proposed method can achieve better or comparable performances than the state-of-the-art method in the category of local methods even with the less running time.In addition,since the depth map is estimated frame by frame,the temporal consistency cannot be guaranteed.This thesis proposes a method to enhance temporal consistency by applying adaptive temporal filtering which explicitly considers reliability of depth and moving attribute of regions.Experiments demonstrate that the proposed depth enhancement algorithm can generate more stable depth sequences and effectively suppress the transient depth errors when rendering virtual images.Due to the inherent drawbacks of stereo matching,the depth map captured by sensor is more robust,especially for the texture-less regions.However,it either suffers from low resolution,or has some holes on the original depth map.Active methods are to solve these problems.Since low-quality depth map always captured with a high-quality RGB color image and they can be registered with each other on the same coordinate system,low-quality depth map can be refined by using the guidance from the high-quality RGB color image.This type of active method is called guided depth map enhancement.The methods on guided depth map enhancement can be classified into different categories depending on whether an external training data is used.Without relying on the external datasets,edge co-occurrence property between depth map and the corresponding color image is explicitly exploited.However,because assumed edge consistency above is not always true,it leads to texture-copy artifacts and blurring depth edges.Markov-Random-Field-based(MRF-based)methods are popular in guided depth map enhancement.The state-of-the-art solutions are to adjust the affinities of the regularization term in MRF energy function.Actually,these existing methods are lack of explicit evaluation model to quantitatively measure the inconsistency between depth edges and color edges,so they cannot adaptively control the efforts of the guidance from the color image.In addition,widely used computing scheme for guidance affinities in the regularization term is based on the depth and color differences between neighbor pixels,which ignores the depth potential structure.In this thesis,three algorithms are proposed to address the problems above.The first one aims to mitigate artifacts caused by edge inconsistency between depth map and color image via hard-decision inconsistency checking pixel by pixel.The second one uses a structural quantitative measurement on edges inconsistency,which is a soft-decision method.It is more accurate than hard-decision counterpart.The third one is to combine such soft-decision edge inconsistency measurement with image potential structure which is modeled on Minimum Spanning Trees(Forest)to acquire more robust depth map enhancement results.These methods are tested on Middlebury and ToF-Mark datasets,which proves progressive improvement.In addition to the handcraft models,data-driven models are expected to implicitly learning such guidance to obtain superior performances.In this thesis,an end-to-end training method based on convolutional neural network is proposed which borrows many concepts from existing models,e.g.batch-normalization and residual learning.It upsamples low-resolution depth map progressively and the residual network is constructed to learn high frequency component in multiple scales.This coarse-to-fine scheme can reconstruct high-resolution depth via multi-frequency synthesis.Experimental results show improvement in subjective evaluation and objective evaluation compared with state-of-the-art methods.