A Study On High Quality Depth Image Accquiring Using Convolutional Neural Network And ToF/Stereo Data Fusion

High quality depth imaging is an important and fundamental problem,different from 2D images,with depth information,more accurate performance can be achieved in many applications such as automatic navigation,virtual/augmented reality,action recognition and robot vision.Up to now,researchers have proposed many depth estimation methods mainly divided into active depth estimation methods and passive depth estimation methods.Active depth estimation includes the use of an active device such as a To F(Time of Flight)camera or a Kinect camera to obtain depth.The passive depth estimation mainly uses the binocular camera for stereo matching so as to obtain the depth of the scene by the estimated disparity.This method have a good performance on textured scenes and can provide high resolution(HR)depth estimation.While it has difficulties to estimate depth in regions with repetitive patterns or smooth area.Active devices such as To F and Kinect cameras perform depth estimation which is independent of surface texture.However,their resolution is low,and they often produce systematic errors including fused depth values in depth discontinuity region,noise caused by low reflectance,flying pixel and so on.Therefore,obtaining high quality depth map can be carried out in three aspects: First,based on the low resolution depth map acquired by To F camera,we can do super resolution and denoising for high quality depth map acquiring.Second,we can design a more robust stereo matching algorithm to solve the problem that traditional stereo matching algorithm is difficult to estimate depth in regions with repetitive patterns or smooth area.Third,using the complementarity of the To F camera and the stereo matching,the To F and the stereo matching data can be fused to produce a high quality depth map.In view of these three ways,based on previous research,we propose our own high quality depth image acquiring methods.We sumarize our contributions as following:1.We propose a high-quality edge map guided super resolution method for single depth image using convolution neural network(CNN).In our framework,we first extract a lowquality edge map from an interpolated depth map.Then we transform the low-quality edge map to a high quality one by our trained deep convolution neural network.To refine the edgs map,we use four edge patterns to connect broken edges and fill holes exist between continuous edges.Guided by the high-quality edge map,we finally utilize a total variation(TV)based model to upsample the initial depth map.Compared with doing texture prediction directly,the edge guidance method has less jagged artifacts and more sharp edges can acquired in the final result.2.We propose a patch-based stereo matching using 3D convolutional neural networks(CNN).Traditional methods often find matching features of a stereo pair based on siamese networks and some other post-processing.However,features extracted by siamese networks are not performance well on depth discontinuity regions.In this paper,we proposed a novel architectures to extract the spatial color features and disparity features together with 3D CNN.The output of our network is binary,we classify each pair to a disparity within one to the maximum of disparities.Finally a new color image guide filter is proposed as a post processing to improve the accuracy of our method.3.We propose variational fusion of Time-of-Flight(To F)and stereo data for depth estimation using edge selective joint filtering(ESJF).ESJF is able to adaptively select edges for depth upsampling from To F depth map,stereo matching-based disparity map,and stereo images.We adopt ESJF to produce high-resolution(HR)depth maps with accurate edge information,and extract a high quality discontinuity maps.We also measure the confidences of To F and stereo data respectively.Finally,we perform variational fusion of To F and stereo depth data guided by the discontinuity maps.Experimental results show that our fusion framework can successfully acquire HR depth maps and has a better performance than thestat-of-the-arts methods in preserving edges and removing noise.