Research On Video Super-resolution Reconstruction Technology

In recent years,with the development of economy and society,the demand for HR(High-Resolution)images and videos has increased unprecedentedly in social production and life.However,due to limitations of imaging device,imaging conditions,economic factors and time,in some cases only LR(Low-Resolution)image sources can be obtained.SR(Super-Resolution)technology,which generates corresponding HR images from input LR images,provides an economical and feasible solution for obtaining HR image sources.Super-resolution technology has made great advances in the past few decades.The recent research on compressed sensing theory and deep learning theory has in-jected new blood into the SR field,which further improves the performance of SR algorithms.Regarding video SR as the main clue,we emphatically study the SR algorithms based on matrix completion and deep learning.The work and innovations of this thesis are summarized as follows:(1)To solve the problem that the video SR algorithms based on matrix completion model,represented by MCSR(Matrix Completion Super-Resolution)algorithm,cannot process the local complex motion in video,a robust video SR algorithm based on matrix completion model is proposed in this thesis.In the motion compensation step,a multi-scale non-local patch matching method is proposed.It exploits the self-similarity to extract enough patches from a few adjacent frames for building the low-rank matrix.The proposed patch matching method not only preserves the edge information effectively,but also improves the robustness to local motion.Experimental results showed that,the proposed patch matching method overcomes the weakness that MCSR algorithm can-not handle videos containing complex motion.In the reconstruction step,a weighted average strategy is designed to obtain HR patches,and reconstruct high resolution video frames more accurately.The simulation results show that,proposed weighted fusion strategy further improves the values of PSNR(Peak Signal to Noise Ratio)by 0.14-0.40dB.(2)Aiming at the problem that SRCNN(Super-Resolution using Convo-lutional Neural Networks)predicts HR images with artifact of ringing on the edges at high magnification,an edge preserving single image SR algorithm based on CNN(Convolutional Neural Networks)is proposed.SRCNN is the first end-to-end CNN SR model.Instead of 9 × 9 and 5 ×5 size of convolutional kernels used in SRCNN,the proposed algorithm applies convolutional kernels with the size of 3×3,which extracts gradient information more effectively.By increasing the depth of model to 6 layers and doubling the number of feature maps,the proposed algorithm predicts HR images with smoother edges.Also a larger training set is used in proposed algorithm to avoid over-fitting.While proposed algorithm has little improvement on training set given by SRCNN,it achieves a better performance on a relatively larger training set extracted from ImageNet.Compared with SRCNN,the increase of PSNR of our algorithm can reach 0.52dB(at a magnification of 2)on average,and no less than 0.08dB(at a magnification of 4),when test on different datasets.Although the increase of PSNR is not obvious at a magnification of 4,the proposed algorithm im-proves the subjective quality significantly for smoother edges and less ringing effect.The proposed algorithm can be applied to predict HR frames from LR video.Although it cannot utilize the information from neighbor LR frames,the complex motion compensation is evaded.(3)We propose a fast structure-preserving video SR algorithm based on 3 D(Three-Dimensional)CNN.Most existing CNN-based video SR algorithms treat the consecutive frames as a series of feature maps,just as the procedure performed in single image SR algorithms.In the proposed algorithm,the input frames are considered as a cube.3D convolution is performed on it to extract features along spatial and temporal dimension.To preserve the structures of reconstructed HR frame,a combination of MSE(Mean Square Error)loss and MS-SSIM(Multi-Scale Structure Similarity Index Measure)loss is used to op-timize the model.Moreover,original LR frames and optical flows estimated by motion compensation sub-net are employed in reconstruction besides the compensated LR frames,providing additional information.The proposed al-gorithm runs fast for that the motion compensation and feature extraction are performed in LR space.The experiment results show that,the values of PSNR and SSIM of the proposed algorithm surpass CNN-based algorithms,such as VSRnet(Video SR Networks)and VESPCN(Video Efficient Sub-Pixel Convo-lutional Network),close to DF(Detail Fusion)algorithm which uses ConvL-STM(Convolutional Long-Short Term Memory network).But the speed of the proposed algorithm is 10 times faster than DF.Compared with the above al-gorithms,the proposed algorithm maintains the structures of the reconstructed image more accurately at high magnification.