Research On Depth Image Enhancement Based On RGB-D Information

Augmented Reality(AR)aims to integrate the virtual objects generated by com-puter technology into the real world in real time.The key point of AR is to under-stand the real world to make the integration of virtual objects and the real world more reasonable from different views,such as geometry,illumination and etc.However,it is essential to obtain much information to understand the real world in AR,and thus makes it a complex problem because it needs many computer vision technologies in-cluding 3D reconstruction,object recognition and so on to solve the problem.Most of the computer vision algorithms use the 2D images which lose much information.Besides,the development of RGB-D cameras(Kinect and RGB-ToF)make it possible to capture depth information of scenes.Depth information describes the distances be-tween objects and depth camera,which provides complementary and effective cues to guarantee the robustness and accuracy in solving the ill-posed problems of computer vision and augmented reality.Generally speaking,complete and high resolution color images of scenes can be obtained easily which are scarcely affected by noise.However,the depth information captured by current depth cameras suffers from many drawbacks,which make it hard to use depth information.Depth image is always noisy and incom-plete,and the depth image always suffers from natural upper limit on spatial resolution.What's more,due to the influence of noise,the registration between depth image and corresponding color image is not accurate enough.Hence,depth image enhancement draws more and more attentions,since the above drawbacks restrict the use of depth information heavily.In this thesis,we provide a comprehensive survey and propose solutions on the state-of-the-art on sub-problems of RGB-D information based depth image enhance-ment,which includes:confidence estimation of depth image,registration of color and depth image of RGB-D cameras and depth image super-resolution.The innovations and contributions of the thesis mainly include:First,proposing an algorithm to estimate per-pixel confidence of depth image cap-tured by RGB-D cameras.Current confidence estimation methods all use laser scanner to provides ground truth,which restrict the use of these methods.Hence,we propose an approach to generate a per-pixel confidence measurement for each depth image captured by RGB-D camera,which does not need the support of laser scanners.We use RGB-D camera to capture a set of depth images of a scene from a fixed view,then we get the average depth image and variance depth image.Variance shows the stability,which can be converted to confidence(ground truth).Meanwhile,various features from both color and depth images are extracted to train depth image estimator using Random Forests regressor.With these learned estimator,a confidence map of any depth image captured by RGB-D camera can be predicted.The experiments demonstrate the effectiveness of our method.Second,proposing an algorithm to register color and depth image of RGB-D cam-eras.A fundamental step to utilize RGB-D cameras is to register color and depth im-ages.However,due to the difference of imaging theory between depth camera and color camera,depth camera can not be calibrated by traditional methods.In this thesis,we propose a method to register color and depth images for RGB-D cameras.A specially-designed checkerboard with hollow squares employed to extract corners and measure camera parameters,which takes advantage of the regularity of corner arrangements and can achieve high accuracy even with noisy depth inputs.Then,we propose a general deviation model to deal with irregular deviations that can not be handled by RGB-D camera projection model.The registration method that incorporates the estimated de-viation model can well register color and depth information.The experimental results demonstrate the effectiveness of the method.Third,proposing two effective network schemes for depth image super-resolution(DSR),including a progressively deep convolutional neural network based DSR and a novel view synthesis based deeply supervised DSR.First,we propose a progressively deep convolutional neural network based DSR.The proposed deep neural network tries to learn the mapping from a low-resolution depth image to a high resolution one in an end-to-end style.Then we exploit the depth field statistics and the local correlation be-tween depth image and color image to better regularize the learned depth map.These priors are integrated in an energy minimization formulation,where the deep neural net-work learns the unary term,the depth field statistics works as global model constraint and the color-depth correlation is utilized to enforce the local structure in depth images.Second,we propose a novel view synthesis based deeply supervised DSR.The pro-posed network describes the task of DSR as a series of novel view synthesis sub-tasks,where each sub-task can be efficiently solved in end-to-end deep convolutional neural network learning.Moreover,the training stage can be conducted in parallel.Further-more,a deeply supervised learning framework is utilized,where strong supervisions are directly applied in different stages.To further exploit the feature maps at different stages,a multi-scale fusion strategy is employed.Experimental results demonstrate the outstanding performance of our methods compared with the state-of-the-art.