Research On Video Stabilization Algorithm With Spatio-Temporal Priori Modeling

With the rapid development of digital imaging and detection equipment,all kinds of hand-held,vehicle-mounted and unmanned aerial cameras have been widely used.Video,as the most vivid expression form of visual information,has become the main way for people to observe and perceive the world.However,due to the lack of stable support during the shooting process,the captured videos may have different levels of jitter,which greatly reduces the viewing experience of people and the performance of subsequent high-level pattern recognition methods.In addition,in the related military fields,the video jitter seriously affects the accuracy of structured video analysis,image interpretation and intelligent decision.Although some optical stabilization devices and professional photographic equipment can be used to compensate the shaky motions during shooting process to achieve the purpose of video stabilization,such kind of method often has high cost and limited stabilization ability.Therefore,the software-based high-performance video stabilization algorithm is the most widely used and powerful stabilization strategy at present.This kind of method can stabilize jitter videos directly without any additional hardware,and also can produce high-quality stabilization results,which has become a hot research topic in the field of video processing.Camera shake can lead to random vibration of video picture,object jumping and deformation,image blur and other degradation phenomena.The purpose of video stabilization is to reconstruct high-quality videos that conform to the photography kinematics rules from low-quality jitter videos.Mathematically,video stabilization is an underdetermined inverse problem of latent smooth motion recovery and image reconstruction in video sequences.In general,video stabilization algorithm consists of three stages: motion estimation,motion smoothing,and image warping.The key point is the solution design of motion smoothing stage.After analyzing the characteristics of camera motion and the shortcomings of existing algorithms,the academic idea of this paper is to excavate the spatio-temporal properties of video inter-frame and intra-frame motion,and deeply study the video stabilization model and efficient algorithm with spatio-temporal prior constraints from the perspective of neighbor motion pattern,motion decomposition and recovery.At the same time,some challenging problems in the stabilization process have also been well considered,such as the smoothing strategy of rapid motion and the rectification of rolling shutter effect.The main contributions and research results of this dissertation are as follows:(1)A novel video stabilization algorithm with total warping variation model is proposed,which effectively reduces the impact of motion estimation error on the stabilization performance.For the conventional algorithms using cascaded motion transformation chain,if the motion estimation is not precise enough,the estimation error will be accumulated due to the multiplication of motion transformations,which could result in the decline of motion compensation accuracy.In order to avoid the accumulative error,a new total warping variation model is proposed in this paper.The model takes the warping transformation of each frame as unknown parameters,and consists of a motion smoothing term and a date fidelity term.After solving the model,the stabilized frames can be generated directly without calculating the motion transformation chain.At the same time,in order to quickly solve the optimization model with highly coupled parameters,the similarity transformation parameters can be divided into additive and multiplicative parameters after a statistical analysis.Thus,solving the whole motion model is separated into independent optimization subproblems for four parameter sequence,and the closed-form solution of each parameter model is given.In addition,an iterative smoothing model for high-intensity stabilization and an online stabilization method for long video sequences are proposed.Experimental results show that the method is robust to motion estimation error and can provide competitive video stabilization results.(2)A locally low-rank regularized video stabilization method with motion diversity constraints is proposed,which fully exploits the properties of neighbor inter-frame motions and first introduces the low-rank prior into motion modeling.Different from many path optimization methods,this method focuses on the relationship between local inter-frame motions.According to the cinematography rules,camera motions in high-quality videos can be divided into three motion patterns: zero velocity,constant velocity,and accelerated motion,and there are only a small number of different motion patterns in a local window.Mathematically,this phenomenon can be expressed as the low-rank property of the motion matrix,so the proposed method introduces the low-rank constraint to smoothed motion for the first time and presents a novel motion smoothing regularization model.In addition,to make the motion smoothing model flexible to different motion conditions,an adaptive motion weighting mechanism with variable length window is proposed.In the data fidelity term,a new motion steering kernel is used to assign autoregressive weight,which shows better performance than classical Gaussian kernel and bilateral kernel.Meanwhile,the adaptive adjustment of local window can further reduce the image content loss caused by smoothing the rapid motion.The experimental results verify the validity of the adopted prior hypotheses and show that the method can provide better motion stability.(3)A novel video stabilization model with motion morphological component priors is proposed.Owing to the multi-component decomposition of inter-frame motion,the limitation of traditional model in dealing with complex motion conditions is overcome.In general,a good stabilization model should have temporal motion adaptability and overall robustness to different motion types.However,most previous methods are prone to produce non-motion adaptive results,such as under-smoothing in low-frequency motion shake segments and over-smoothing in rapid motion segments.In order to solve these problems,this method decomposes the observed inter-frame motion into three motion morphological components: low-frequency smoothed motion,high-frequency compensatory motion,and noisy motion.By integrating the spatio-temporal characteristics of different motion patterns,and combining with weighted kernel norm,local autoregression and sparse constraint,a joint optimization model with motion morphological component priors is established,and an efficient stabilization algorithm is designed based on the Alternating Direction Method of Multipliers(ADMM)framework.Meanwhile,by detecting and segmenting the rapid motion segments,a dynamic adjustment method of regularization parameters is constructed,which realizes the flexibility of stabilization algorithm for various motion patterns.Experimental results show that this method can generate high-quality stabilization results in different kinds of motion types.(4)A simultaneous video stabilization and rolling shutter removal method via joint inter-frame and intra-frame motion modeling is proposed,which solves the problem that stabilization alone cannot effectively eliminate the rolling shutter effect.Due to the row-wise exposure mode of CMOS camera,video jitter and rolling shutter effect often exist at the same time,so only solving one of them is hard to obtain a good visual effect.Therefore,the method explicitly models and estimates the mesh-based inter-frame and intra-frame motions,and uses the spatio-temporal relationship between the two kinds of motions to directly calculate the warping transformations.Firstly,in order to accurately estimate the mesh-based inter-frame motions,a new neighbor motion consistency term is introduced and the adaptive weight calculation method is given.Secondly,different from the single motion assumption adopted by the traditional roller shutter removal methods,this method calculates the spatially-variant intra-frame motions,and thus the rectified motion can more truly reflect the impact of depth change.Finally,the calculation process of motion-aware warping transformation is presented,which flexibly adjusts the stabilization intensity for different motion conditions.The experimental results show that the method achieves the best comprehensive performance among several quantitative metrics.