Depth Map Joint Up-Sampling Algorithm Based On Convolutional Neural Network

Nowadays,depth map has played a more and more important role in computer vision tasks,such as auto-driving,3D modeling and human machine interacting.However the resolution of depth map provided by commercial depth cameras can not satisfy the demand of production.Hence the depth map super-resolution algorithms arise.Different from traditional images,depth map has a lower signal to noise ratio,less channels and semantic information.Thus its super-resolution process is more difficult.However,Depth map's super-resolution with a guidance image usually has a better performance,and it is cheap and convenient for modern technology to produce a depth map accompanied with an auxiliary image,so it is practical and meaningful to develop joint depth map super-resolution with guidance images.Deep learning has achieved great performance in traditional image super-resolution and joint depth map super-resolution researches.State-of-the-art methods use convolutional neutral networks to extract features from depth map and guidance image for reconstruction,which effectively avoid “texture transferring”.However,state-of-the-art methods mainly have two drawbacks.Firstly,these methods only utilize the high dimensional features from the bottom layers,ignoring the low dimensional features from the shallow layers,nor the advantages brought by multi-scale receptive fields.Secondly,state-of-the-art methods usually up-sample the depth map to the desired resolution in the preprocessing process,or up-sample the depth map 2 times bigger in each super-resolution stage.Such up-sampling scales are too big for the low quality depth maps,causing the up-sampled depth maps blurred,not aligned with the guidance images,failure of repairing of the network.This paper proposed two methods to deal with the problems respectively.The main contributions include the following aspects:1.To deal with the first problem.We propose Dense Deep Joint Up-sampling Network.The network uses small cascaded convolutional kernels to enlarge receptive field,and uses skip connections to help the information flow through the deep network and collect the low dimensional feature in the shallow layers.The skip connections form different paths with multi scales,alleviating the gradient descending problem.Extensive experiments were performed on Middlebury,SUN-RGBD,Lu and NYUv2 datasets.Quantitative and qualitative comparisons with state-of-the-art methods showed the effectiveness of the proposed method.2.To deal with the second problem.We proposed a Laplace pyramid structure network called Mini-Step Joint Up-sampling Network.It uses a scaling factor smaller than 2 to progressively up-sample the depth map and utilize more guidance information.Experiments on Middlebury 2001,2003 and 2005 datasets were performed.Quantitative and qualitative comparisons with state-of-the-art methods showed the superiority of the proposed method.