Research On Multi-frame Image Super-resolution Algorithm

The goal of image super-resolution algorithms is to reconstruct the high-frequency information that is lost in low-resolution images,then generate corresponding high-resolution images,and finally achieve the purpose of improving the visual effect of images.Recently,the rapid development of deep learning technologies has made single image superresolution algorithms increasingly mature.With the development of photography and storage technologies,the storage method of image information has been evolved from single image to multi-frame image(such as stereo images,short videos,long videos,etc.).Compared with single image,multi-frame image store more information and are more significant for medical,transportation,and other fields.However,multi-frame image superresolution reconstruction algorithms are still in theirs infancy,and the performance and efficiency of image reconstruction need to be improved.This thesis focuses on the task of multi-frame image super-resolution and addresses two typical problems: stereo image(two frames)and video(multiple frames)super-resolution.During the shooting of stereo images or videos,adjacent frames have high similarity but also contain certain displacements and visual differences due to subtle differences in shooting time and space.The information in multiple frames can assist each frame to obtain richer details,edges,and high-frequency information,thus enabling the reconstruction of more accurate high-resolution images.However,due to the existence of pixel offset,directly stacking different frames will cause blurring and ghosting.Thus,the core difficulty of multi-frame image super-resolution lies in accurately aligning and fusing multiple frames of information.For strero image super-resolution tasks,most of the existing algorithms use an attention mechanism to fuse left and right view information,but only at spatial dimension,ignoring the influence of the channel dimension,thereby limiting the performance of reconstruction.Aiming at this problem,this thesis proposes a lightweight parallax multidimensional attention module(PMDA),which corrects the pixel differences inside the two views based on the multi-dimensional attention mechanism,ensuring that the complementary information between the two views can be fully and correctly fused to improve the super-resolution of strero images.In addition,this thesis also proposes a general strero image super-resolution framework,which can embed any single image super-resolution model based on PMDA to facilitate the development of strero image super-resolution.Relevant experiments prove the excellent effect of this module and framework.For video super-resolution tasks,most methods would aggregate information in different frames through optical flow estimation and motion compensation.Although these methods have achieved confident results,it is difficult to achieve a balance between model performance and complexity.This thesis also proposes a new efficient spatio-temporal network,which is designed to separately encode the spatial and temporal information of video frames through two parallel streams.Furthermore,to better adaptively align the features of these two streams at the feature level,this thesis proposes a gated control module composed of deformable convolutions to fuse the spatial and temporal features.Based on this design,not only efficient spatio-temporal feature mining can be achieved,but also a lightweight model can be obtained.The effectiveness of the proposed method is demonstrated through both quantitative and qualitative results.Finally,based on the existing problems in the current research,this thesis discusses the future research and innovation directions of multi-frame image super-resolution.