System Design And Deep Learning-based Reconstruction For Snapshot Compressive Imaging

Multi-dimensional imaging is an image acquisition technology which can sense and measure the multi-dimensional information of scenes beyond two-dimensional spatial imformation.Three-dimensional space,one-dimensional time,and the physical quantity of wavelight,such as spectral,phase and polarization etc.contain vast amounts of information that human eyes can not fully perceive.These information not only have wide applications in remote sensing,medical and food safety,but also play an important role in the field of computer vision and pattern recognition,which makes multi-dimensional imaging technology become the key technology to realize the future intelligent society of everything interconnection.Due to the limitation of traditional imaging theory and sensor hardware,multi-dimensional imaging systems usually use multiple cameras or mechanical scanning to capture the images,which suffer slow speed,high system complexity and high cost.Thanks to the development of computational imaging and compressed sensing theory,snapshot compressive imaging(SCI)technology based on hardware coding and software decoding is proposed.SCI systems uses a two-dimensional detector to capture multi-dimensional data in a snapshot measurement,which greatly improves the acquisition speed and reduces the system complexity.However,for the image reconstruction task which is a key of SCI system,the traditional optimization algorithm suffers from the slow reconstruction speed and poor reconstruction quality,which seriously limit the application of SCI systems.To overcome the above problems and challenges,based on compressed sensing and deep learning,this dissertation deeply investigates the applications of SCI systems and high-speed and high-quality image reconstruction methods.The specific research contents and innovations are as follows:1.We proposed a snapshot compressive multispectral endomicroscopy system which combines fiber bundle imaging,SCI system and deep learning algorithm to realize the acquisition and reconstruction of multispectral endoscopy images with cellular resolution and 24 spectral bands in video frame rate,which is conducive to the formation of a practical fast and low-cost medical endoscopic diagnosis tool.In addition,a single disperser spectral SCI system and a video SCI system based on digital micromirror are built.The former can capture 28 spectral band images at the range of 450-650nm,while the latter can collect dynamic scenes using a low-cost camera with 10-30 times frame rate,providing reliable datasets for algorithm researches.2.Focusing on deep learning-based reconstruction methods,this paper makes innovative research on network structures,interpretability,flexibility and generalization,and proposes a variety of reconstruction methods based on deep learning for SCI systems.Compared with the existing methods,the proposed methods can provide the state-of-the-art results.1)In order to overcome the limitations of traditional optimization algorithm in SCI reconstruction,we propose three end-to-end neural network structures.For spectral SCI,we propose a spatial-spectral self-attention mechanism to learn the spatial and spectral correlation by generating the attention maps at each dimension separately.This mothod solves the problems of huge memory usage and high computation redundancy of the original self-attention for image tasks.Secondly,we propose the recurrent bottleneck and multiscale residual convolution to enhance the ability of feature extraction and modeling the channel relationships.For video SCI,we propose a deep neural network structure based on residual network to improve the convergence ability of the network.The end-to-end neural network can reconstruct the experimental measurements in 100 milliseconds level,which allows an end-to-end closed-loop sensing system with the ability of real-time acquisition and reconstruction,and thus premote the industrialization of SCI system.2)To solve the problems of weak interpretability and low flexibility of end-to-end neural network,we combine the optimization algorithm and deep learning,and propose a deep unfolding neural network,which integrate the convolutional neural networks into an unfolded optimization iteration process to form a cascade network and conduct end-to-end training.This method can be used in spectral SCI and video SCI systems flexibly,and is insensitive to the masks,and achieves the state-of-the-art performance in speed and quality of reconstruction.3)In order to solve the problem of low generalization caused by the shortage of datasets in supervised learning,we propose a self-supervised neural network framework for spectral SCI.This method uses an untrained neural network to learn the prior of spectral images directly from the compressive measurements.By combining with the plug-and-play framework,our method achieves high-quality spectral image reconstruction without any training data.This method has potential advantages in SCI applications lacking multi-dimensional image datasets.