Video Super-resolution Reconstruction Research Based On Deep And Shallow Networks

The goal of super-resolution reconstruction is to recover the corresponding highresolution image or video from the low-resolution image or video.As a typical computer vision problem,super-resolution reconstruction has been studied for decades.In recent years,the popularity of HD display promotes the development of video super-resolution reconstruction.Compared with image,the difficulty of video super-resolution reconstruction lies in the highly similar time dimension,which leads to the increase of the steps of motion estimation and adjacent frame information fusion.This paper focuses on the video superresolution reconstruction method based on deep learning and explicit motion estimation,and proposes a deep and shallow video super-resolution network with attention mechanism(ADS-VSR)and an improved video super-resolution network with high-resolution optical flow estimation(OFE&ADS-VSR).The main work of this article can be summarized into three points:(1)First of all,this paper learns the representative single image and video superresolution reconstruction methods,as well as optical flow-based motion estimation methods,and then implements and compares several representative methods.(2)In this paper,a deep and shallow video super-resolution network with attention mechanism(ADS-VSR)is proposed and applied to video super-resolution reconstruction.ADS-VSR network consists of a shallow network which can reconstruct the basic information of video frames,a deeper network which can reconstruct the high-frequency details,and an attention module based on attention mechanism.Joint training of ADS-VSR network can effectively improve the fitting ability and convergence speed of the model.The process of video super-resolution reconstruction using ADS-VSR network is as follows: firstly,the low-resolution video frames are successively motion estimated and motion compensated by the optical flow estimation method with local-global smoothness assumption and total variation regularization constraint and the adaptive motion compensation method.Then the compensated low resolution video frames are sent to ADSVSR network to output high resolution video frames.Experimental results show that the video reconstructed by ADS-VSR network contains more detailed information and achieves better visual quality.(3)In this paper,an improved video super-resolution network with high-resolution optical flow estimation(OFE&ADS-VSR)is proposed to realize end-to-end video superresolution reconstruction.OFE&ADS-VSR network has two characteristics: a)optical flow estimation network(OFE)based on pyramid structure,which estimates the potential highresolution optical flow between the corresponding high-resolution video frames from coarse to fine for the input low-resolution video frames,b)ADS-VSR network based on post up sampling mode,which effectively improves the network training efficiency.The realization of video super-resolution through the end-to-end OFE&ADS-VSR network can be summarized into three stages: I)The low-resolution video frames are sent to OFE network to obtain high-resolution optical flow,and the connection between high-resolution optical flow and low-resolution video frames is established by using the conversion of space to depth to generate the corresponding low-resolution optical flow set,II)the motion compensation of the adjacent frames is made by using the optical flow value in the low resolution optical flow set,and the multi low resolution adjacent frames after compensation and the center frame to be reconstructed are combined into the low resolution draft set,III)the low resolution draft set is sent to the ADS-VSR network in the post upper sampling mode,and the high resolution result of the center frame is output.A large number of experimental results show that OFE&ADS-VSR network can further improve the quality of video reconstruction.Through the quantitative and qualitative comparison on vid4,harmonic-8 and SPMC-11 test sets,it can be concluded that the reconstruction results of the method proposed in this paper are closer to the real high-resolution video frame,contain more detailed information,and give people a more pleasant visual experience.