Research On Video Stabilization Based On Temporal-Spatial Optimization

In recent years,with the popularization and development of intelligent devices,capturing videos by portable cameras has become an important way to record everyday lives.In addition,cameras are also used to perform some complex tasks,such as track-ing,recognition and so on.However,due to the limitations of conditions or external scenes,cameras often suffer the lack of professional stabilizating instruments,which often causes severe jitters in the captured videos.These jitters will not only cause the degradation of video quality,but also affect the processing of subsequent tasks based on these unstable videos.So video stabilization is an essential research direction.At present,stabilization for common videos has been well solved,but the existing algorithms often suffer from the performance degradation or even failure when dealing with videos containing complicated scenes.These complicated scenes include,for ex-ample,large foreground moving objects,strong parallax and discontinuous depth vari-ation caused by the occlusion of dynamic foreground objects.This paper analyzes the above challenges and proposes several novel methods to resolve them.We bulid sev-eral optimization functions containing temporal constraints and spatial constraints.The main work and contributions of this thesis can be summarized as follows:1)A video stabilization method based on both foreground and background feature trajectories is proposed.This method mainly stabilizes unstable videos including large foreground objects and strong parallax.The most typical videos are traffic videos.Traf-fic videos are the videos captured by cameras mounted on vehicle and are used to record traffic situations,so there are often large foreground objects such as vehicles and strong parallax in these videos.Unlike most of the traditional methods which only use back-ground feature trajectories for camera motion estimation and smoothing,this method no longer distinguishes foreground and background feature trajectories and uses all of them to estimate the camera motion.By solving a well-designed optimization problem,we can eliminate the high-frequency component of the camera motion,that is,the camera jitter,and then stabilize the video.Because this method makes use of both foreground and background feature trajectories,it is better than those methods only consider the background feature trajectories,especially when the foreground object is large and the number of extracted background feature trajectories is not enough.Furthermore,some refinements are proposed to speed up our method and enhance its robustness.2)A content-aware video stabilization method based on adaptive blocking strat-egy is proposed.Previous works usually take feature trajectories in the background to estimate one global transformation or several transformation matrices based on fixed meshes,and warp the shaky frames into their stabilized views.However,these methods can not model real-world motion well in complicated scenes,such as scenes containing large foreground objects or strong parallax,and may result in notable visual artifacts in the stabilized videos.To resolve the above issue,this thesis proposes a novel method which stabilizes the shaky video based on all of its contents and an adaptive blocking strategy.More specifically,this method first extracts feature trajectories of the shaky video and then generates a triangle mesh according to the distribution of the feature trajectories in each frame.Then we design a two-stage optimization problem based on these triangle meshes to perform the estimation and smoothing of the camera motion.To further enhance the robustness of our method,we propose two adaptive weighting mechanisms to improve its spatial and temporal adaptability.3)A video stabilization method based on the learning of pixel-wise warping maps is proposed.Traditional video stabilization methods usually produce assignable errors during handling videos containing occlusion and depth variation,such as foreground occlusion and parallax variation.It is also difficult for these methods to stabilize low-quality videos,such as night-scene,blurry,noise,watermarked videos and so on.This thesis proposes a neural network for pixel-wise warping mapping estimation,which is called" PWStableNet".The outputs of the network are two warping maps of the same size as the video frame.The stabilized frame is generated by these warping maps.The effective training of the network is realized by designing an appropriate loss function.The proposed method is built upon a multi-stage cascade encoder-decoder architecture which enables the latter stage to learn the residual from the feature maps of former stages and achieve better performance at latter stages.This is the first deep learning based pixel-wise video stabilization method.This method has faster processing speed than the traditional methods,and a better performance can be expected through training with more videos in various situations.