Video Denoising Based On Noise Robust Motion Estimation

Although image sensor technology has developed rapidly in recent years,video signals are often subject to multiple types of noise during acquisition or transmission.,producing unpleasant visual effects and affecting other areas of video analytics,including Target detection,target recognition,motion tracking and semantic segmentation.Video data has significant temporal redundancy compared to still image data,enabling still image data to provide richer scene content.Therefore,more information can be utilized in video denoising than image denoising.Although video denoising algorithms have made great progress in recent years,there are two prominent problems in general: 1.The optical flow field estimation in the noise video is not accurate.The effect of video denoising depends first on the accuracy of optical flow estimation.Using optical flow information,different video frames can be calibrated,and then a denoising algorithm is applied to further eliminate noise.However,traditional optical flow estimation methods often have problems such as unclear boundaries,poor use of time continuity information,and insufficient robustness to noise,which cannot meet the requirements of video denoising;2.Most existing video denoising algorithms assume that video noise is additive white Gaussian noise,which is often impractical for dealing with real-world mixed noise.In the actual noise environment,Gaussian,Poisson and impulse noise are often included at the same time,and a more reasonable model needs to be designed to deal with the mixed noise.Aiming at the problems existing in the existing methods,this paper firstly explores the noise-robust optical flow estimation method,and then proposes a new denoising algorithm based on this.The research results of this paper include:(1)The first special end-to-end depth learning method motion boundary detection is proposed,which is called motion boundary network.We introduce a refined network structure that uses the source image,the initial optical flow,and the corresponding transform errors as inputs to produce high-resolution motion boundaries.We designed our network as a codec network,in which the decoding network adopts a cascade structure of multiple sub-networks,and adds residual connections to achieve fast propagation of gradients,and enables end-to-end training to be performed more efficiently.The proposed network is capable of exploring multiple levels of image and motion features and is robust to initial optical flow estimation errors.Based on the motion boundary detection,we further optimize the initial optical flow with the obtained motion boundary.We designed a fusion network of motion boundary and optical flow,and guided the optimization of optical flow with motion boundary to further remove the noise in the initial optical flow result and make the final optical flow boundary clearer.(2)A multi-frame optical flow estimation model based on low rank sparse optimization theory is proposed.The model makes full use of the context time domain continuity of video image sequences,as well as the low rank of multi-frame optical flow and the sparsity of moving targets.Multi-frame optical flow estimation is performed in noisy video.Based on the traditional method,we add a new improvement to the noise robustness,and propose a multi-frame optical flow solution for the noise environment to solve the objective function.The model includes color consistent term,trajectory low rank term,edge smoothness penalty term and robustness.Connection items,etc.,can effectively model multi-frame optical flow in a noisy environment.We use the convex optimization theory based on low rank sparse to solve the model,and decompose the solution process of the multi-frame optical flow estimation model into two sub-problems.Comparative experiments on multi-frame optical flow estimation show that the algorithm in this chapter can achieve better results in both qualitative and quantitative aspects.(3)A robust video denoising algorithm is proposed to remove mixed Gaussian,Poisson and impulse noise in video data.Firstly,based on the optical flow estimation in the previous two chapters,the noise-robust similar image block grouping is carried out,and then the mixed noise video denoising problem is transformed into a robust low rank and sparse minimization problem.We design and embed an effective prior model in this framework to deal with Gauss-Poisson noise components in mixed noise.The proposed algorithm can effectively remove video noise in a mixed Gaussian-Poisson-pulse noise environment.Experiments show that the proposed video denoising method achieves satisfactory performance and is superior to the previous method in both qualitative and quantitative methods.(4)A multi-scale recursive residual convolutional neural network is proposed.We first apply the image block matching strategy described in Chapter 4 to the reference frame space based on the optical flow estimation methods in Chapters 2 and 3.Then,the denoising model is obtained through end-to-end training to denoise the video image in the mixed noise environment.The multi-scale recursive residual network proposed in this chapter learns the general and detailed features of noise video images from the global and local levels through a multi-scale network architecture.Then,the residual unit structure is combined with the recursive unit structure,and the recursive residual units are stacked to form a sub-network structure of each resolution level.In the proposed recursive residual unit,the internal connection structure shares the same input,and there are multiple paths between the input and the output,which is beneficial for learning more complex features.In order to adapt to the mixed noise environment,we designed a multi-scale L1-L2 norm loss function to supervise the training process of the entire network.