Video Compressive Sensing With Frame Interpolation Network

In the Information Age,as an important part of getting information,the sampling of analog signal to digital signal is limited to Nyquist Sampling Theorem,which means higher information acquisition,storage and transmission cost.The breakthrough of compressive sensing means that lower sampling frequency can achieve the same result as high sampling frequency does by some way.The traditional compressive sensing method solves the optimization problem to get expected result,which often takes long time and is difficult to be applied to real-time scene.Data-driven deep learning method spends most time training and runs fast when actually used.Deep neural network usually has a large number of parameters,not appropriate for edge devices,so network compression is necessary.Quantization method is well known for its ability to reduce model size largely with little accuracy loss,so it is applied to compressive sensing deep neural network in this paper.In video compressive sensing field,different methods have to measure every single frame,which means higher cost under high frame rate scene.To solve this problem,optical flow based frame interpolation and compressive sensing are combined together.Our main work is listed as follows.For the problem of redundant parameters not being good for edge devices,we apply quantization method which is usually used for high level tasks like classification on compressive sensing deep network.Quantization in this paper is binarization,which is an extreme way of quantization.Float parameter is quantized to two values,which means the storage cost is reduced from 32 bit to 1bit,much less than before.We have two different experiments about quantization,one is 0 or 1 binarization for measurement network the other is 1 or-1 binarization for residual blocks of reconstruction network.During the process of 0 or 1 binarization,we found it is hard to train which means the result is far from satisfactory,so a new network much easier to train is proposed and the new network is proved to be theoretically equal with the original network.During the process of 1 or-1 binarization,we found the result is not good enough,so scaling factors are used to improve the reconstruction result.In video compressive sensing field,almost every video compressive sensing method measures every single frame,resulting in high cost under high frame rate scene.To solve this problem,optical flow based frame interpolation and compressive sensing are combined together in this paper.The proposed method measures part of frames and synthesize frames which are not measured with the method of optical flow based frame interpolation,it helps lower cost.Since part of the frames are measured,the measurement rate of each frame can be increased,which provides a good basis for frame interpolation.Optical flow based frame interpolation consists of optical flow prediction and frame synthesis.In the optical flow prediction phase,inspired by the image pyramid in the traditional optical flow prediction method,the more robust multilayer features are used to predict the optical flow layer by layer,leading to more accurate result.In the intermediate frame synthesis phase,the intermediate frame is obtained based on the linear motion assumption,which is often correct under high frame rate scene.Compared to other methods,our method achieves better result under the same measurement rate.In all,we quantify compressive sensing deep networks for the problem of large number of parameters,then for the high cost problem of video compressed sensing under high rate frame rate scene,we creatively combine optical flow based frame interpolation and compressive sensing to get better result.