Temporally Coherent Video Matting And Composition

Image/video matting refers to the problem of accurately extracting foreground objects from images or videos.It is a fundamental operation of image/video editing.It is widely used in a variety of fields,such as photo editing,film and animation production,and virtual reality.Compared to image matting,video matting should ensure to generate temporally coherent mattes because the human visual system is highly sensitive to temporal jitter and flickering.In addition,video matting has to face larger scale of data,complex and various backgrounds which may consume more computational resources and involve more interactions.This paper proposes a video matting framework combining key frame extraction,optical flow propagation,and Bayesian matting.Concretely,it first makes use of inter-frame difference to adaptively select the key frames of a video.The trimaps of key frames are then generated semi-automatically by using the GrabCut image segmentation.These trimaps are propagated to intermediate frames by using the bidirectional optical flow technique.Color and gradient information are further exploited to rectify the correspondence error in this process.Then,the Bayesian matting is adapted to video matting by considering volumetric color sampling to make full use of temporal information,and introducing 3D ? regularization constraint to preserve temporal coherence of ? values and reduce temporal jitter and flickering.Finally,we implement a prototype system of video matting and synthesis to verify the proposed framework.A variety of tools are provided for interactive matting,background replacement and new video synthesis in this system.Experiments demonstrate that our method can obtain spatial accurate and temporal coherent results with high efficiency.