Multi-scale Mutual Feature Convolutional Neural Network For Depth Image Enhancement

With the development of the depth camera technology,we can obtain depth images more and more conveniently.In many fields such as virtual reality and 3D reconstruction,depth information plays a significant role.However,there are two problems with depth images captured by consumer-level depth cameras.One problem is that depth images are often disturbed by noise and there are large black holes caused by the missing depth values.Another problem is that the depth image has a low resolution and cannot provide high-accuracy depth information.The above problems limit the development of related applications based on depth information.Hence,depth enhancement is a problem that needs to be solved urgently.To solve these problems,we focus on depth enhancement in this thesis,including depth denoise and depth super-resolution.The innovations and contributions of the thesis mainly include:According to the characteristics of depth image noise,we propose a data-driven depth enhancement method based on mutual feature multi-scale convolutional neural network.This end-to-end framework has two stages,i.e.,mutual feature learning and depth regeneration.For multi-scale mutual feature learning,we designed two multi-scale mutual feature generators to extract more reliable mutual features.Then,depth transformation sub-network combines two mutual features to recover clean depth image.We tested our model on several well-known datasets in terms of PSNR and visual effect.Compared with state-of-art algorithms,our method provides better performance of these two criteria and also has a higher processing speed.In addition,our model also has strong generalization performance and can handle the noise of the depth image in the real environment.For depth image super-resolution reconstruction,we propose two convolutional neural networks based on multi-scale mutual features,including a depth super-resolution deep network based on multi-scale mutual feature guidance(MSMF-DSR)and a depth superresolution network based on multi-scale mutual features and residual fusion(RF-MSMF-DSR).To avoid interference of redundant information in color images,MSMF-DSR uses multi-scale mutual features as guide information to enhance the effect of depth super-resolution reconstruction.Because low-resolution depth images and high-resolution depth images have common low-frequency information,RF-MSMF-DSR adopts residual learning to convert the learning target of the network into a nonlinear mapping of low-resolution input to residuals.In addition,the deep network and shallow network are combined to realize the residual fusion of different levels of residual components.In this way,the RF-MSMF-DSR has achieved better performance and better generalization.The experimental results show the effectiveness of the above two methods in terms of RMSE and visual effect.