Research And Application Of Spatiotemporal Video Saliency

With the development of science and technology, people have achieved significant progress in gathering visual information, video display, storage and transmission. However, the attendant problem is that the task of real-time video processing becomes particularly difficult to overcome. Video saliency is born to solve the above problem and has become a red-hot research. Today, a significant number of video saliency detection models are based on the pixel domain. The current video saliency detection model must first decode the compressed video to spatial space, and then adopt feature extraction. For the video decoding process, it is not only time consuming but also complicated to calculate. In recent years, video processing based on compressed domain has gained increasing attention. This is because video processing using the stream information has no need to fully or partly decode the video, which benefit for real-time video processing. Currently, video saliency detection based on compressed domain is in early research stage, and it needs to be improved significantly in terms of detection accuracy. Thus, the further study of spatiotemporal video saliency detection based on compressed domain is of great significance.Through the study of existing video saliency detection model, this paper has proposed a video saliency detection model considering the drawbacks of the existing compress-based video saliency detection model. Firstly, by extracting the sequence of H.264compressed video stream information, we get spatiotemporal characteristics, such as color, texture, luminance and motion vectors. Then, we normalize the motion vector in temporal domain and estimate the motion vector using global motion compensation. Lastly, we calculate the video saliency based on the above features. The method can be divided into the following three steps: spatial video saliency detection, temporal video saliency detection and the combination of spatial video saliency and temporal video saliency. Experiments show that the proposed video saliency detection method based on video compressed domain performs well in both subjective and objective experiments. This paper also proposes a video fingerprint extraction technique based on video saliency through the study of existing video fingerprinting technology. In the video fingerprint extraction stage, we extend the video saliency as the basic unit for video fingerprint extraction to get the robust digest of spatial video fingerprinting. In addition, by clustering the video fingerprinting in the same shot, we get the robust digest of temporal video fingerprinting. In the video fingerprint matching stage, the existing matching algorithm has been improved by adjusting the decision threshold and has improved the accuracy of video matching. Experiments show that the proposed saliency-based video fingerprinting, compared with existing video fingerprinting, has lower computational complexity, and has stronger robustness to spatial video processing and temporal video processing.