Deep-Learning Based Reconstruction Algorithms For Three-Dimensional Computational Imaging Systems

Computational imaging combines computer vision,signal processing methodologies,and classical imaging systems,aims to reconstruct high-dimensional continuous signals by designing new imaging mechanisms,new imaging systems,and new reconstruction methods.In particular,3D reconstruction is one of the research hotspots in computational imaging.This paper focus on designing depth reconstruction algorithms for three novel 3D imaging techniques(light field imaging,photon-efficient imaging,and non-line-of-sight imaging).Existing deep-learning(DL)based methods have poor generalization capabilities in depth reconstruction for the above three imaging techniques.In order to settle the problem,this paper designs accurate imaging models,introduces new learning strategies and network architectures,and exploits the correlations in the measurements.Finally,DL based reconstruction methods are designed with superior generalization capabilities and reconstruction performance than existing stateof-the-art approaches,and also facilitate extra real-world applications.The details are listed below:Light field depth reconstruction with zero-shot learning.Existing DL based methods rely on training pairs from a certain specific camera model,which can hardly generalize to any other camera models.In order to settle the problem,we introduce the zero-shot learning for the first time and design a light-weight network to fully exploit the spatial-angular-temporal correlations in the single light field data for depth reconstruction.Specifically,we design a compliance loss and a divergence loss so that the network can be trained on single light field data without additional training samples.Further,we extend the network for light field video depth reconstruction by exploiting the extra temporal correlations for the first time.Compared with the existing approaches,the proposed method not only has superior reconstruction performance,but also can generalize to any light field cameras.In addition,the method further facilitates downstream applications,e.g.,light field super-resolution and view synthesis.Photon-efficient depth reconstruction with a non-local neural network.Existing DL based methods can hardly generalize to data with low photon counts,low signalto-background ratio(SBR),and multiple returns.In order to address the problem,we build a forward model considering low photon counts,low SBR,and multiple returns all together for the first time.We deduce that the detection time for noise photons is uniformly distributed,and demonstrate that there exist long-range spatial-temporal correlations in photon-efficient measurements.Thus,we design a multi-branch framework with one encoder and two decoders for photon-efficient depth reconstruction.The encoder is based on a non-local neural network to extract long-range spatial-temporal correlations from the measurements,while the depth decoder and the intensity decoder are adopted to recover both depth maps and intensity images,respectively.Compared with the existing approaches,the proposed method not only has superior reconstruction performance,but also can generalize well to real-world data captured with different imaging systems.In addition,the method facilitates the long-distance imaging techniques over 20 kilometers away.Non-line-of-sight(NLOS)depth reconstruction with Transformer.The complicated imaging process hinders the existing DL based methods in generalization to realworld measurements.In order to address the problem,we build the forward model considering both noise photons and detection properties of sensors,and analyze that there exist complementary local and global correlations in NLOS measurements.Thus,we introduce Transformer that has powerful modeling capabilities,for NLOS depth reconstruction,and design a reconstruction network which contains a multi-scale local block(MSLB),a multi-scale global block(MSGB),and a local-global integration block(LGIB).The MSLB and MSGB are adopted to extract multi-scale local and global information from NLOS measurements,while LGIB integrates the local and global information in a token space with a multi-scale manner.Finally,shallow features and deep features are further integrated with each other to generate depth maps.Compared with the existing approaches,the proposed method not only has superior reconstruction performance,but also can generalize well to real-world measurements and reconstruct large-scale complicated scenarios.In addition,the method promotes downstream applications,e.g.,NLOS object classification.In summary,this paper focuses on depth reconstruction for light field imaging,photon-efficient imaging,and NLOS imaging,by designing new DL based methods with superior reconstruction performance,generalization capability,and practicability over the existing approaches.