Video Sequence Alignment Based On The Combination Of Movement Information And Background Information

Video sequence alignment is an important field of image processing. It is a technology that temporally and spatially aligns unsynchronized video sequences derived from different viewpoints or different time. It has been widely applied to many fields, such as video surveillance, target identification, video mosaic and three-dimensional reconstruction. The main work and contributions of the thesis are as follows:Firstly, two traditional trajectory-based video sequence alignment methods are discussed in detail. One is based on the time-line constraint and the other is based on the projective invariant representation. Previous studies have shown that there are some problems with these available methods. The time-line constraint based alignment method adopts the epipolar geometry information between background images to describe the trajectory points, and utilizes the intersection points of epipolar lines and trajectories to find candidate matching trajectory points. This method performs well when the trajectories in the video sequences are curve. But it cannot work well for some special video sequences, in which the trajectories are approximately straight lines and coincide with their epipolar lines, since there are too many intersection points of epipolar lines and trajectories. The projective invariant representation based alignment method adopts the shape information within neighborhood to describe the trajectory points, and utilizes the statistical properties of the cross ratio to find candidate matching trajectory points. Then the time synchronization of video sequences is realized in this way. But a potential problem may arise in scenes where the trajectories have many segments with similar shape features. In this situation, the method cannot differentiate between trajectory segments, so that it will lead to a great alignment result error. Furthermore, the above two methods ignore the spatial relationship between moving objects and background features when describing the trajectory points.Secondly, in allusion to the problems in the above two methods, a novel video sequence alignment algorithm is proposed based on the combination of movement information and background information. In the proposed algorithm, the background images and trajectories of the input video sequences are extracted respectively by the low-rank and sparse matrix decomposition algorithm, and the matching feature point pairs between the background images are obtained by a feature point detecting and matching algorithm.Next, the fundamental matrix between the background images is estimated by the matching feature point pairs, and four feature point pairs which are not coplanar in 3D physical world are selected to build a coordinate system. Then, the spatial relationship between moving objects and background features is employed to further improve the distinctness of each trajectory point. The trajectory points are described and matched by both the coordinate system and epipolar geometry constraint to obtain the candidate time point pairs. Finally, the time line is fitted by the random sample consensus algorithm, and the space-time transform model parameters of the video sequences are obtained. Several sets of experiments demonstrate that the proposed algorithm outperforms some traditional methods, such as the time-line constraint based and the projective invariant representation based ones, in terms of alignment accuracy. Especially, the proposed algorithm still works well for video sequences, in which the trajectories are approximately straight lines and coincide with their epipolar lines or have many segments with similar shape features.