Research On Surveillance Video Cropping Methods

Modern video surveillance system, i.e., network video surveillance system, consists of networks of cameras connected to a central control room which includes a collection of monitors. We are interested in the design of the future control room, and envision an architecture consisting of a large display wall which acts as a single entity, as opposed to matrices of independent monitors, as in current state of the art control rooms. The display wall allows operators to flexibly assign variable areas and locations to the available videos from surveillance cameras. Now following problems come up: we need to build a central control system to determine what video to display, where to display it and how to do that. This paper describes only the video cropping component of the system and helps to focus the attention of operators on specific parts of the scene.We consider the problem of cropping surveillance videos as choosing a trajectory that a small sub-window can take through the video, selecting the most important parts of the video for display on a smaller area. We model the information content of the video by whether the image changes at each pixel. Then we show that we can find the globally optimal trajectory for a cropping window by using a shortest path algorithm. In practice, we speed up this process without affecting the results, by stitching together trajectories computed over short intervals. After that, we show that using a second shortest path formulation can help us find good cuts from one trajectory to another, improving coverage of interesting events in the video. We have applied the proposed algorithm to the surveillance video datasets provided by PETS 2006.We first present the problem definition of video cropping and then describe the fives main stages of the algorithm in detail. Assume the input video segment has T frames. Each frame t can be covered by a set of n variable size overlapping windows. These windows are labeled Wi ,t, with i being the window number, selected from an index set I ? ?1 ,2,???,n?. Define the cross product setΩ? I?I?????I?IT. Then the problem is formulated as: where i ? I, Q ?? is the window sequence that maximizes the saliency, C ??? is a cost function that decays with increasing saliency, d (? ,?) is a distance measure and A( W) is the area of window W . To guarantee a smooth path, windows in two consecutive frames are restricted to be close to one another and with little area variation.The five main stages of our algorithm are listed as follows:1. Motion energy is used as a measure of the"saliency"of a cropping window trajectory. It is efficiently computed in real time, and captures important activity in surveillance video. We compute frame differences and threshold them to detect motion, then apply morphological operations to the resulting motion frames. The motion energy is computed to be the number of 1's in the resulting binary images. The remainder of the algorithm works on these preprocessed difference frames.2. The video is modeled as a graph of windows, with its edge weights reflecting saliency captured by windows, efficiently computed using integral images.3. A shortest path algorithm finds the window trajectory that captures the overall maximum motion energy. The resulting trajectory is smoothed to remove jiggles and staircase-like appearance. This procedure is repeated several times on the remaining parts of the video to capture the remaining saliency. This results in obtaining a set of disjoint smooth paths of cropping windows that capture as much saliency as possible.4. A secondary optimization procedure produces the final path by alternately jumping between the paths computed earlier, selecting which one to follow at which time, so as to maximize both captured saliency and covered regions of the original video.5. Long videos are processed by breaking them into manageable sub-videos, while allowing overlap between consecutive subvideos, to produce smooth transitions.We are mainly interested in surveillance applications. Typically, much more video than operators can observe is available. In addition, this video is unedited, and more importantly, not focused on any agent in the scene. This has made our approach different from earlier approaches that dealt with edited videos, such as movies, news reports, or classroom video. In particular, the proposed method offers the following contributions:1. The size of the cropping window can be variable.2. We can use multiple cropping windows to cover more activities in the scene.3. Only a relatively short video segment needs to be processed at one time, not the complete video, which makes the algorithm an online algorithm.4. By stitching together results from short segments of a video, we get a result identical to the globally optimal one, given the entire video.