Optimization-based summarization and indexing of extended videos, with application to instructional video semantics

We develop and demonstrate methods to summarize and index extended videos in a semantically informed way, and present semantic compression and linking techniques for instructional videos. We also demonstrate for the first time the extreme degrees to which instructional videos can be usefully compressed and indexed.; We first define and design a generic uniform approach for the selection of key frame subsets from videos. By converting the problem of video summarization into the problem of recognizing those key frame subsequences that optimize pre-defined criteria, we present novel off-line optimization-based approaches, and evaluate their superior results compared to those of existing algorithms. Two separate video summarization semantic criteria are provided and explored, one more favorable for video content summarization, and the other more favorable for network transmission and reconstruction. We further extend off-line optimization approaches to real-time on-line dynamic semantic compression by using a human memory buffer model of time-constrained video perception. This model provides a dynamically ratio-adjustable semantic compression for video summarization and streaming. An entire hierarchy of key frame subsets can be retrieved at no additional cost. We show the relationship between the off-line model and the on-line model, and analyze their performance results.; Next, we show that these methods naturally lead to efficient summarization of instructional videos by defining semantic measures that capture the content of this genre. Focusing on two dominant presentation formats of instructional videos, those of hand written slide and blackboard, we provide a rule-based approach for extracting clean summary frames with significant content. We demonstrate the extraction of “content panorama slides” that summarize hand-written content in these videos, by extracting and stitching together their content while maintaining the qualitative spatial relationship of the text lines and figures. We also introduce a novel concept of “semantic teaching unit”, a meaningful spatial-temporal unit in instructional videos, and present a rule-based method to extract them. We create from semantic teaching units a highly a compact summary (temporally compressed several thousand times), enabling reconstruction and indexing of instructional video content. Finally, we demonstrate how videos containing presentations using formatted computer slides can be efficiently recognized and indexed to their computer-readable sources.