Research On Blind Motion Video Deblurring Based On Deformable Convolution

Blurred video restoration has always been an important topic in the field of computer vision and digital image processing,and its research is significant for daily life,security,medical treatment,space exploration,etc.In recent years,the convolutional neural network has been widely used in blurred video restoration.The restoration methods based on the convolutional neural network need to effectively aggregate complementary information between video frames,which contains the clear texture details of the reference frame and is the key to achieve high-quality restoration of the target fuzzy frame.Recent studies show that accurate video frame alignment can effectively improve the network's learning ability of complementary information between video frames,thus improving the restoration quality of motion blurred video.However,the complex motion in video results leads to local inconsistent inter-frame content dislocations and complex non-uniform motion blur,which hinder the effective and accurate alignment of video frames.To solve this problem,this thesis studies the blind restoration method of motion blurred video based on deformable convolution to achieve accurate frame alignment,to make full use of the complementary information between frames,and improve the restoration quality of motion blurred video.The main work of this thesis is as follows.(1)A multi-scale deformable convolutional method for blind motion video deblurring(MDVD)is proposed.Firstly,MDVD takes the two-stage strategy of pre-alignment and3 D compensation to align video frames step by step.In the pre-alignment stage,a multiscale deformable convolution alignment module is designed based on multi-scale and cascade,which aligns video frame features from coarse to fine.In the 3D compensation stage,the 3Dconv module is designed based on 3D convolution.And the decoupled channel,temporal,and spatial attention layer CTS is designed to optimize the 3Dconv module by giving more weight to the clearer and sharper content of the complementary information between frames.This two-stage video frame alignment strategy reduces the frame alignment pressure of each stage of the network,and can effectively deal with large-scale and serious inter-frame content dislocations,thus reducing the difficulty of network training and convergence,and improving the accuracy of frame alignment.In addition,a spatial stream is designed in this method.The spatial stream is composed of two kinds of wide activation residual blocks.By changing the spatial resolution or channel of the active layer feature,the spatial information of the target frame,such as texture details,can be enriched without increasing the parameters.The design of spatial stream further improves the restoration quality of motion blurred video.In particular,MDVD(w/o spatial stream)without spatial stream still has high restoration quality.However,the proposed method has the following problems: 1)The inter-frame content dislocations and non-uniform motion blur caused by complex motion in the video make it difficult for the designed multi-scale deformable convolution alignment module to train and optimize,thus affecting the accuracy of video frame alignment;2)The model size and time consumption of the proposed method increase significantly.(2)A patch-based multi-scale deformable convolutional method for blind motion video deblurring(PDVD)is proposed for overcoming the shortages of MDVD in(1).Firstly,aiming at the difficulty of training and optimizing the multi-scale deformable convolution alignment module,the patch-based deformable convolution alignment module is designed.This module transforms the alignment of frame features into the alignment of feature patches with different scales and partial overlaps.Because the complex motion and non-uniform motion blur of frame features are simplified to the single motion and uniform blur of feature patches,the alignment of feature patches is easier to train and converge than the alignment of frame features.So the video frame alignment based on the feature patch is more accurate.Then,aiming at the problem that the model size and time consumption of the method in(1)increase significantly,this method omits the spatial stream in(1),removes the attention module in the 3dconv compensation module,and reduces the use of the 3D convolution layer.After the improvement of the network structure,the model size and time consumption of the method are effectively reduced.To verify the proposed MDVD and PDVD,the ablation and contrast experiments are carried out on two typical blurred video datasets.Experimental results show that for motion blurred video,the restoration quality of MDVD,MDVD(w/o spatial stream),and PDVD is better than most existing methods,especially MDVD.However,in terms of model size and time consumption,PDVD is lighter and more efficient than most current restoration methods.Besides,due to the improvement of frame alignment accuracy of patch-based deformable convolution alignment module,compared with MDVD(w/o spatial stream)in(1),the model size and time consumption of PDVD are reduced by 68%and 71%,respectively,while the restoration quality of motion blurred video is reduced by less than 1%.