Content-aware Video Object Synopsis And Editing

With the rapid development of digital media technique, the video data captured by digital cameras, network cameras, cell phones, PDAs, and video surveillance cameras has obtained explosive growth, leading to a large-scale video database. Effective synopsis and editing of the important objects in those videos is a frontier research topic in computer graphics and computer vision. The aim of video object synopsis and editing is to preserve the important objects while removing redundancies in the source video, producing a much shorter synopsis video for the source one. Besides, it also supports editing operations such as video object copying, pasting, removing, and completion. Video object synopsis and editing has wide applications in the fast video briefing, video storing, network video transmission, video retrieval, video sharing and disseminating, video gaming and entertainment.There exist many challenges when developing an effective and fast video synopsis and editing system. For example, when synopsizing video objects in the temporal domain, only shifting the objects along the time axis will result in serious collisions and occlusions between the objects which affect the user’s accurate observation and comprehension of the video content. When resizing the video in the spatial domain, it should not only preserve the important regions, but also should avoid the excessive stretching and deforming of the unimportant regions. Otherwise, apparent distorting artifacts will occur in the less important regions when there exist not much texture or too many structures. Furthermore, previous methods cannot handle the redundancies existing in the movements of a video object, and also there is no method that can demonstrate the synopsis video in an all-round, multi-view, and interactive way.Focusing on the above key issues, this paper conducts a more systematic study of the video object synopsis and editing problem from multiple different research views. Firstly, we study the temporal synopsis and editing of multiple video objects. Then, we discuss the spatial synopsis and editing of video content. After that, we further give an in-depth study of the synopsis and editing problem of a single video object’s activities. At last, this paper synthesize novel views for the synopsized and edited video to show the video in an all-round and interactive way.In summary, the contributions of this paper are:(1) We propose a temporal video object synopsis and editing method based on object spatiotemporal optimization, which shift the video objects in both spatial and tem-poral domains to increase the motion space for video objects. By object spatiotem-poral optimization model, we not only produce a more compact synopsis video but also solve the problem of serious collision and occlusion artifacts between object-s. Furthermore, we propose a multi-level image patch relocation method based on Markov Random Field (MRF) which synthesizes a more compact background that can match the shifted objects.(2) We study the spatial video content synopsis and editing method by combining warp-ing and summarization operations. The proposed method can effectively preserve the important video objects while avoiding excessive distorting of the unimpor-tant regions, thus can produce visually pleasing synopsis and editing results. The method first takes a feature-preserving warping method to pre-process the input video, and obtain an initial synopsis and editing result. Then a synthesis-based summarizing method is used to optimize the initial result, which can reduce the distorting artifacts in the initial result caused by warping method. The method also supplies other video object editing operations such as video object completion.(3) We propose an object movements synopsis and editing method based on object part assembling and stitching optimizations, to eliminate the redundancies existing in the moving actions of object. Firstly, a MRF-based object part assembling optimization is given, which can effectively select the important actions of object parts, and assemble them together. Then, an object part stitching optimization is presented to stitch the assembled object parts seamlessly. After two optimizations, the final object with synopsized and edited movements is obtained.(4) We construct for the synopsized video an multi-view browsing and navigation sys-tem based on novel view synthesis, solving the problem of showing or playing a synopsized and edited video in an all-round and interactive way. Under a specified view, the proposed projection occlusion elimination method can avoid the projection occlusions between objects. We also provide automatic viewing angle adjustment method according to video content that can help user to always stay in the optimal viewing angle along with the video playing.The four main contents of this paper all concentrate around the topic of video object synopsis and editing, which are closely linked to each other, and thoroughly solve the problems. The above methods are integrated into a unified framework for video object synopsis, editing, and multi-view browsing. The experiments on various different kinds of videos demonstrate the effectiveness of our methods. Our methods can also be used in the applications of augmented reality, video retrieval, video synopsis and resizing, intelligence city, movie editing, and surveillance video processing, etc.