Video Fusion Based On Low-rank And Sparse Matrix Decomposition

In recent years, with the rapid development of sensor technologies, video sensors have been widely employed in video surveillance systems and machine vision fields. In real applications, in order to get the complete information of a scene, multiple video sensors are simultaneously employed to capture the content of the same scene. To utilize the information captured from multiple video sensors sufficiently and efficiently, we need to combine the contents from different video sensors into a fused video. This can be easily realized by using video fusion, which merges multiple videos from different video sensors into a composite video containing important infromation from different video sensors that can discribe the scene more accurately.The main work and contributions of the thesis are as follows:Firstly, the thesis discusses several commonly-used video fusion algorithms in detail, including video fusion algorithms based on the spatial-temporal energy matching, the spatial-temporal structure tensor, the pulse coupled neural network and the higher order singular value decomposition. Previous studies have shown that there are some problems in these available algorithms. Even though the algorithm based on the spatial-temporal energy matching fuses the videos as a whole and improves the temporal consistency, it treats the spatial and temporal information contained in the input videos equally and merges them by the same fusion strategy, which decreases its spatial-temporal consistency to some extent. The algorithm based on spatial-temporal structure tensor uses different fusion strategies for the spatial and temporal information and produces better performance, but it gets the improvement at the cost of high computational complexity. Moreover, the above algorithms can hardly obtain satisfying results when the videos are noisy. The algorithm based on pulse coupled neural network can obtain a relatively satisfying result when the input videos have high signal to noise ratio(SNR), but it has high computational complexity and doesn’t work well when the input videos have low SNR. The algorithm based on higher order singular value decomposition has high computational efficiency and good fusion performance under noisy circumstances, but it employs the same feature image to describe the background information with HOSVD, which decreases its spatial-temporal consistency to some extent.Secondly, a novel video fusion algorithm based on the 3D Surfacelet Transform(3D-ST) and the low-rank and sparse matrix decomposition(i.e., the robust pricinple component analysis, RPCA) is proposed to solve the problems in the existing algorithms. The proposed algorithm consists of the following procedures:(1)Input videos are decomposed into lowpass subbands and bandpass subbands using the 3D-ST;(2)The bandpass subbands are then decomposed into low-rank components and sparse components by using the RPCA. Specificly, the low-rank components contain the spatial information in the scene backgrounds and the sparse components contain the temporal information related to moving objects;(3)The lowpass subband coefficients, the low-rank components and the sparse components are merged with different fusion schemes, respectively;(4)The fused video is constructed by performing the inverse 3D-ST on the merged subband coefficients.Finally, several sets of experiments demonstrate that the proposed algorithm performs better in spatial-temporal information extraction as well as in spatial-temporal consistency than some of the existing video fusion methods.