Video Fusion Performance Assessment Based On Spatial-temporal Salience Detection

Recently, video fusion has been widely applied in many fields, such as remote sensing, video surveillance and military. As more and more video fusion methods have been developed over these years, video fusion performance evaluation is becoming increasingly important. The evaluation criteria can help people choose the best fusion method and search the optimal parameters.The main work and contributions of the thesis are as follows:1. The thesis introduces several popolar image and video fusion performance evaluation algorithms, and all of these algorithms are analyzed in detail. Studies show that most of the existing video fusion performance algorithms are designed for noise-free input videos, few researches have been done on the video fusion performance for noisy input videos. While in real world applications, videos are vulnerable to random noise in the process of video capturing or transmitting. In recent years, some methods have been proposed to fuse noisy videos, in which the extraction of important spatial-temporal information and the noise suppression are simultaneously achieved. When the input videos are noisy, most of the existing video fusion evaluation methods mistake noise as important spatial-temporal information, and they will mistake some useful spatial-temporal information as being lost if a video fusion method has suppressed the noise in input videos, which will provide some evaluation results that are inconsistent or even in conflict with the subjective evaluation results.2. A novel video fusion performance evaluation algorithm based on spatial-temporal salience detection is proposed to address the problems of the existing video fusion performance evaluation methods mentioned above. A spatial-temporal salience detection algorithm based on tri-dimensional spatial-temporal structure tensor is used to divide the input videos and the fused video into two types of regions: spatial-temporal salient regions(mainly containing spatial-temporal salient information) and noisy regions(mainly containing noise). A spatial-temporal information extraction metric based on the maximum eigenvalue of the tri-dimensional spatial-temporal structure tensor is designed for spatial-temporal salient regions to evaluate how well the spatial-temporal information from input videos is extracted. A noise suppression metric based on the minimum eigenvalue of the tri-dimensional spatial-temporal structuretensor is designed for noisy regions to evaluate how well the noise from input videos is suppressed. Finally, the global fusion quality metric is constructed by combining the spatial-temporal information extraction metric and the noise suppression metric. Several sets of experiments demonstrate that the evaluation results of the proposed global metric coincide with the subjective results well when input videos are noisy. In addition, the spatial-temporal information extraction metric can be directly used to evaluate different fusion methods for noise-free input videos.