A Video Saliency Detection Method Based On Frequency Domain Prior And Spatiotemporal Correlation

With the development of deep learning and the significant effects of neural networks in computer vision tasks,deep learning is widely used in computer vision tasks such as object tracking,object detection,pedestrian re-recognition and image retrieval.Saliency detection has become a research hotspot as a data preprocessing technique in other computer vision tasks,and with the popularity of video equipment,people are facing more dynamic scenes,so video saliency detection has become a popular research direction.The purpose of video saliency detection task is to detect the continuous moving saliency target which can attract the attention of human eyes through a specific algorithm.In this paper,video saliency detection model is divided into two modules: static module and dynamic module.Firstly,the saliency detection of static image based on prior knowledge in frequency domain is carried out.Then,based on the static module,the temporal-spatial features of the video frame are fused by the dynamic saliency detection module to get the final video saliency detection result.Specifically,the main work of this paper is as follows:(1)Frequency-domain visual saliency analysis of video frames by hypercomplex Fourier transform.By extracting the quaternion feature of video frame,constructing quaternion image of video frame,and performing quaternion Fourier transform to analyze the frequency-domain saliency of video frame and get the prior knowledge of video frame in frequency-domain,which is used as the input of static image saliency detection module;(2)construct static image saliency detection module based on prior knowledge of frequency-domain,and on the basis of the static module,explore the effectiveness of the prior knowledge obtained from the hypercomplex Fourier transform of multiscale spatial analysis for image.The static module adopts the structure of symmetric convolution,that is,there are sibing branches of the network input,which take the current video frame?its reciprocal image and the prior knowledge of frequency domain as the network input,the network weight of Image Net pre-training is used to initialize the network parameters of static module by using the transfer learning based on shared parameters.Then tested on the public dataset of the image and video saliency detection,and compared with the static image saliency detection algorithms without prior knowledge of frequency domain qualitatively and quantitatively;(3)Construction of dynamic video saliency detection module based on temporalspatial feature fusion.Firstly,the static saliency detection results corresponding to the current video frame are obtained through the static module;Then,through optical flow estimation method flownet2.0,the optical flow of the current video frame is calculated by using two adjacent video frames,and the motion feature of the current video frame are calculated by using the optical flow.Finally,the current video frame,its corresponding static saliency detection results and the motion feature of the current video frame are taken as the network input of the dynamic module to carry out the video saliency detection based on the temporal-spatial feature fusion.The dynamic module adopts the structure of full convolution neural network,and uses vgg-16 as the basic network model.The network initial parameters of the dynamic module is initialized with the network parameters pre-trained by Image Net in the way of transfer learning based on shared parameters.And compared with the existing video saliency detection algorithms in the public dataset of video saliency detection qualitatively and quantitatively.