Research On High Precision Depth Map Super-Resolution Reconstruction

Depth map super-resolution(DSR)is the task of recovering a high-resolution(HR)depth map from its low-resolution(LR)version,which is of great significance for practical applications of depth maps,such as 3D reconstruction,posture estimating and autonomous navigation.However,due to the imaging limitation of depth sensors in real conditions,high quality and high resolution(HR)depth maps are often difficult or even impossible to be acquired directly,thus effective pro-processing DSR techniques are needed to yield HR output from the degraded LR counterpart.To improve the performance and practicability of depth reconstruction,we proposed two DSR frameworks.As we observe in the study,DSR task face many challenges.First,depth boundaries and details are generally hard to reconstruct due to the limited information in LR space.Second,depth regions on fine structures and tiny objects in the scene are destroyed seriously.To tackle these difficulties,we propose a progressive multi-branch aggregation network(PMBANet),which consists of stacked multi-branch aggregation(MBA)blocks to progressively recover the degraded depth map.Specifically,each MBA block has multiple parallel branches: 1)The reconstruction branch is proposed based on the designed attention-based error feed-forward/-back modules,which gradually highlight the high-frequency informative features at depth boundaries.2)We formulate a separate guidance branch to help to recover the depth details,in which the multi-scale branch is to learn a multi-scale representation that pays close attention at objects of different scales,while the color branch regularizes the depth map by using auxiliary color information.Then,a fusion block is introduced to adaptively fuse and select the discriminative features from all the branches.The extensive experiments on benchmark datasets demonstrate that our method achieves superior performance in comparison with the state-ofthe-art methods.Although existing color-guided DSR methods have demonstrated remarkable progress,several limitations still remain.First,these methods require paired RGB-D data as training examples to jointly recover the degraded depth map.However,the paired data may be limited or expensive to be collected in actual testing environment.Second,considering the memory consumption and computing burden,the processing on the HR RGB data also hinders the practical application.Motivated by the above analysis,we explore for the first time to learn the cross-modality knowledge at training stage,where both RGB and depth modalities are available,but test on the target dataset,where only single depth modality exists.We propose a cross-task interaction module to realize bilateral knowledge transfer scheme in a teacher-student roleexchanging fashion and an extra structure regularization to learn more informative structure representations for depth recovery.In the actual deployment and testing,the proposed method has the characteristics of lightweight and faster speed,and can achieve excellent performance even without the assistance of high-resolution color information.