Joint Color Image And Depth Map Super-resolution Using Convolutional Neural Network

The task of a qualitative increase in image resolution is one of the most current issues of digital image processing.Super-resolution(SR)methods are the group of signal-processing algorithms,which allow producing a high-resolution(HR)image from single or multiple low-resolution(LR)images of the same scene.Not long ago,deep neural networks(DNN)have been introduced to such fields as computer vision,machine translation,natural language processing,speech and audio recognition,social network analysis,bioinformatics,medical image analysis and material inspection.Convolutional neural network(CNN)also has been widely applied to color image and depth map super-resolution problem,where a high-resolution depth map can be restored from a LR depth map with the guidance of an additional HR or LR color image of the same scene.Proposed method is based on the algorithm,that HR depth map is reconstructed by joint LR depth map and corresponding LR intensity image.The Joint double branch network(JDBNet)is formed with a multi-scale upsampling conception for solving image super-resolution problems.Such approach can considerably enhance the condition of the recovered HR depth images.Network subdivided into two independent networks – JDBNet1 and JDBNet2.The main difference between JDBNet1 and JDBNet2 is that JDBNet2 has two mean squared error loss function as the last output layer in intensity Y-branch and the last output layer in depth map D-branch.This,in turn,allows JDBNet2 outperform JDBNet1.Low-resolution intensity image and lowresolution depth map of the same scene are input data for training networks.The output data of the system is a high-resolution depth map.These networks were trained for upscaling factor 2×,4×,8× and 16×.Experiments show that proposed networks perform much better than the modern advanced depth image super-resolution methods,conducted by other authors.The performance of represented methods was evaluated by Root Mean Square Error(RMSE).