Object-Based Keyframe Extraction For Surveillance Video

To achieve efficient video representation and fast access, video key-frame abstraction technology, which focuses on extracting the key information from original video and reducing the redundancy of video based on video content analysis, emerges. Furthermore, the video key-frame extraction technology can also reduce the storage space of the video. Surveillance video is captured with statically equipped camera which has limited apparent structure characteristics and contains a lot of redundant content such as identical background. To solve these problems, we propose two methods of video key-frame extraction for surveillance video. One is object-relation-based key-frame extraction method and the other one is object-based key-frame concertation method. Compared with the existing methods, our methods are more effective and robust.In this paper, the main work and contributions are:(1) Inter-frame object correlation based dissimilarity measurement:on the basis of object-based frame representation, we define a new inter-frame dissimilarity measurement associated with object links. Combining with object class label, the inter-frame object correlation can be calculated by matching distinct object sets detecting from different video frames. As a result, dissimilarity between different frames depends on the object sets correlation. Hence, matching proportion between two object sets indicates the inter-frame object correlation. The higher matching proportion, the stronger the object correlation and the smaller the dissimilarity between these two frames, and vice versa. That is to said, our keyframe extraction method based on object links considers the inter-frame object correlation, which helps dividing an original video into visually coherent video chunks, where each chunk contains independent object set, expressing coherent visual contents of this chunk. Surveillance key-frame extraction based on object links can extract a compact keyframe group to represent more significant information of original video.(2) Attention rule based key-frame selection: video partition process decompose an original video into visually coherent video chunks consisting of consecutive video frames. For each video chunk, we present a new key-frame selection way combined with attention rule, which is formulated as key-frame selection saliency function. The medoid method selects a corresponding frame denoting the feature center of this video chunk as key-frame, while the middle method selects a middle frame of this video chunk as key-frame. According to the semantic gap between low-level feature description and high-level concept, it is difficult for these two methods to select robust keyframe from each video chunk. Therefore, we propose two standards including Content Completeness and Visual Satisfaction to measure an effective keyframe. By fusing of two standards as a saliency function, we could get the most satisfied keyframe corresponding to the highest saliency score. As a result, our novel keyframe selection method based on this score function can select effective key-frames from video chunks robustly.(3) Compact key-frame formulation for surveillance video:to solve the problems of video synopsis existing, the change of video sequence, pseudo collision, redundant background and low compact degree of video abstraction, we further enrichment the traditional video keyframes to get more compact video abstraction, named compact keyframe, we model the compact keyframe extraction problem as a maximum a posteriori estimation optimization problem, which can be further solved by using Markov Random Field (MRF). Combining with the Linear Programming Relaxation (RLP) process, all object locations in compact key-frames could be determined by all of these object trajectory optimization in original surveillance video. These object locations integrated with the optimal position corresponding to each object area can generate a compact key-frame to express significant contents of an original video. Generally speaking, this paper proposes concept of compact keyframe concentrate, and formulates this definition into a constraint optimization problem.