Research And Development Of Real-Time Video Super-Resolution Algorithm Based On Embedded GPU

Video super resolution is an image processing technology that converts a lowresolution video stream into a high-resolution video stream.Compared with traditional algorithms,current video super-resolution algorithms based on deep learning achieve state-of-the-art performance.But these algorithms have a large number of model parameters.Even on high-performance servers,these algorithms can hardly achieve real-time forward inference,which makes it difficult to be widely used in actual scenarios.In order to reduce the number of model parameters of video super-resolution algorithms and accelerate the forward inference speed of the network,this thesis studies on the real-time video super-resolution algorithm.The main contents are as follows:(1)This thesis develops a video super-resolution algorithm based on 3D convolution,which solves the problem that algorithms with small parameters cannot effectively deal with the large-scale motion in videos.Firstly,a multi-scale deformable convolution structure is used to estimate the motions of adjacent frames with respect to the center frame,and then a spatio-temporal fusion structure based on 3D convolution is developed to fuse feature maps at different moments.Later on,a depth-wise convolution scheme is adopted in network design to reduce model parameters of the3 D convolution kernel.Lastly,a reconstruction structure with attention mechanism is designed to focus on important feature channels.Experimental results show that the algorithm can perform forward inference on videos at a faster speed while maintaining a relatively high reconstruction metrics.The PSNR and SSIM values on the REDS4 datasets are 27.69 dB and 0.8055.(2)In order to improve the computing performance of the video super-resolution algorithm based on 3D convolution,this thesis designs a lightweight video superresolution algorithm by optimizing the network structure on the basis of the algorithm.And then,a knowledge distillation method for video super-resolution algorithms is designed to improve the reconstruction performance of the lightweight network.The experimental result shows that the model trained with the knowledge distillation method can achieve a 0.45 dB improvement in reconstruction metrics.Among three algorithms with parameters less than 2M compared in this thesis,the lightweight video super-resolution algorithm has the best PSNR and SSIM values on the REDS4 datasets,reaching 26.89 dB and 0.7725 respectively.And the lightweight video super-resolution algorithm can perform forward inference at 34.76 fps on a server with Nvidia 1080 Ti.On the Vimeo-90 k datasets,the PSNR and SSIM values of lightweight video superresolution algorithm are 33.52 dB and 0.9128 respectively,close to the effect of mainstream video super-resolution algorithms.And the algorithm can perform forward inference at 27.55 fps on the embedded GPU Xavier.