A Study On Depth Image Enhancement Based On Co-occurrence Of RGBD Images

With the technology of deep cameras and related equipment continues to change,progress and mature,we can more easily and quickly obtain the depth information of the scene.In many tasks of computer vision,depth information has a wide range of appli-cations,such as human-computer interaction,autonomous driving,3D reconstruction,and virtual reality.Recently,especially the rise of RGBD cameras has brought new development directions and research hot spots to the acquisition of depth images.How-ever,the depth images captured by such consumer cameras are often low-resolution,with many holes and noise,etc.These problems can seriously hinder the application of depth images in related tasks.Therefore,how to enhance the quality of depth images is the most urgent problem to be solved.In this context,this paper has conducted in-depth research on the problem of deep image enhancement.The main research work and innovations are as follows:First off,this paper present an explicit interpretation of the co-occurrence of depth and color image based on intrinsic image decomposition theory.From the two perspec-tives of gradient domain and appearance domain,the co-occurrence between RGBD images is analyzed and explained,and the reason,why the texture information on the color image will be introduced to the depth image during the depth image enhancement process,is explained.Then,based on the above analysis,a framework based on RGBD image co-occurrence is proposed for depth image enhancement.In addition,in order to locate the salient edge between depth and color image,this paper proposed a structure edge detection algorithm based on convolutional neural network(CNN).Firstly,based on the VGG network,this paper makes full use of the different scale features of VGG's convolutional layer output to predict the edge of the depth image,respectively.Then,the edge detection results at different scales are first merged.Next,it is supervised to predict the depth structure edge and thus the fusion result is obtained.By combining the features obtained by convolutional layers of dif-ferent scales,the optimal edge prediction results at each scale can be obtained.Finally,through a large number of experiments on the NYU v2 dataset and the comparison with some advanced edge detection algorithms,the effectiveness of the proposed edge de-tection results of this paper is demonstrated.Finally,this paper proposed a CNN-based framework to learn the co-occurrence of RGBD images for guided depth enhancement.The framework consists of three parts:single depth image enhancement,structure depth edge detection,and guided depth en-hancement.Firstly,a low quality depth image is input to a single depth image enhance-ment network for preprocessing.Next,the color image and the enhanced depth image are input to the previously proposed structural edge extraction network to detect the depth structure edge.Then,the obtained depth structure edge,the color image,and the enhanced depth image are input to the guided depth enhancement network,and the co-occurrence of the RGBD image is learned and it effectively enhance the depth image.Finally,the proposed framework is applied to many depth image enhancement experi-ments including denoising,super-resolution,and restoration.The experimental results show that the proposed algorithm can effectively learn the co-occurrence of RGBD im-ages to enhance the depth image.