Deep Compressed Sensing For Image And Video

The emerging technology of compressed sensing(CS)depicts a new paradigm for image acquisition and reconstruction that implements the sampling and compression processes jointly.More specifically,the CS theory shows that a signal can be recovered from many fewer measurements than suggested by the Nyquist-Shannon sampling theorem when the signal is sparse in some domain.The CS theory has been a hot research topic since it was proposed,and it has brought great changes to the field of signal processing and communication.After more than a decade of development,the CS technology has been further improved,such as distributed CS and Bayesian CS have been developed.The CS technology has also been applied in many fields,such as magnetic resonance imaging and wireless broadcast.However,despite the tremendous development of the CS technology in theory and application,the research on the CS technology has encountered a bottleneck in recent years.The traditional sparsity-regularized-based CS methods either have poor signal reconstruction quality or have high computational complexity.Therefore,it is an urgent task to study the new CS methods with low complexity but high reconstruction quality.In recent years,deep learning has achieved breakthrough applications in various fields,and it is possible to obtain good performance with low computational complexity.In this paper,we use deep learning to solve the challenge problems in the study of CS,and propose deep compressed sensing(Deep CS)for image and video.Specifically,this study mainly includes the following four aspects:First,a novel image compressed sensing framework using convolutional neural network(dubbed CSNet)is proposed.In the study of compressed sensing,the two main challenges are the design of sampling matrix and the development of reconstruction method.On the one hand,the usually used random sampling matrices are signal independent,which ignore the characteristics of the signal.On the other hand,the state-of-the-art image CS methods achieve quite good performance,but with much higher computational complexity.To deal with the two challenges,CSNet jointly optimizes a sampling network and a reconstruction network.The sampling network adaptively learns the sampling matrix from the training images,which makes the CS measurements retain more image structural information for better reconstruction.Specifically,three types of sampling matrices are learned,i.e.floating-point matrix,{0,1}-binary matrix,and {-1,+1}-bipolar matrix.Thelast two matrices are specially designed for easy storage and hardware implementation.The reconstruction network,which contains a linear initial reconstruction network and a non-linear deep reconstruction network,learns an end-to-end mapping between the CS measurements and the reconstructed images.Experimental results demonstrate that CSNet offers state-of-the-art reconstruction quality,while achieving fast running speed.In addition,CSNet with {0,1}-binary matrix,and {-1,+1}-bipolar matrix gets comparable performance with the existing deep learning based CS methods,and outperforms the traditional CS methods.What's more,the experimental results further suggest that the learned sampling matrices can improve the traditional image CS reconstruction methods significantly.Second,a scalable convolutional neural network for image compressed sensing(dubbed SCSNet)is proposed.SCSNet is the scalable extension version of CSNet,and it achieves scalable sampling and scalable reconstruction with only one model.Specifically,SCSNet provides both coarse and fine granular scalability.For coarse granular scalability,SCSNet is designed as a single sampling matrix plus a hierarchical reconstruction network that contains a base layer plus multiple enhancement layers.The base layer provides the basic reconstruction quality,while the enhancement layers reference the lower reconstruction layers and gradually improve the reconstruction quality.For fine granular scalability,SCSNet achieves sampling and reconstruction at any sampling ratio by using a greedy method to select the measurement bases.Compared with the existing deep learning based image CS methods,SCSNet achieves scalable sampling and quality scalable reconstruction at any sampling ratio with only one model.Experimental results demonstrate that SCSNet has the state-of-the-art performance while maintaining a comparable running speed with the existing deep learning based image CS methods.Third,an iterative convolutional neural network for color image compressed sensing(dubbed ICSNet)is proposed.ICSNet jointly learns a sparse sampling matrix and an iterative multi-channel joint reconstruction network.The learned sparse sampling matrix not only reduces both the amount of computation and the need for storage space,but also makes ICSNet can be better applied to compressed imaging of color images.The iterative reconstruction network performs multi-channel joint reconstruction,which makes use of the relevant information between color image channels to improve the quality of reconstructed images.Furthermore,ICSNet is a lightweight iterative network,which can significantly improve the quality of reconstructed images through the multiple iterationsof the trained iterator.Experimental results show that ICSNet has the state-of-the-art performance while maintaining a comparable running speed with the existing deep learning based image CS methods.Fourth,a novel video compressed sensing framework using convolutional neural network(dubbed VCSNet)is proposed.VCSNet is an extension of the first three works in the video domain,and it makes use of both intra-frame and inter-frame correlations to improve the reconstructed video quality.Specifically,VCSNet divides the video sequence into multiple groups of pictures,of which the first frame is a key frame that will be sampled at a higher sampling ratio than the other non-key frames.In a group of pictures,the block-based frame-wise sampling by a convolution layer is proposed,which leads to the sampling matrix being automatically optimized.In the reconstruction process,the framewise initial reconstruction by using a linear convolutional neural network is first presented,which effectively utilizes the intra-frame correlation,then the deep reconstruction with multi-level feature compensation is proposed,which compensates the non-key frames with the key frame in a multi-level feature compensation manner.Such a multi-level feature compensation allows the network to better explore both intra-frame and interframe correlation.Extensive experiments on six benchmark videos show that VCSNet with the multi-level feature compensation provides superior performance over the stateof-the-art video CS methods and the deep learning based image CS methods in both objective and subjective reconstruction quality.