Collaborative Manifold Ranking For RGB-T Visual Saliency Detection

With the development of science and technology,mass data have been produced,which enrich people's daily life,but bring great challenges to the computer vision tasks at the same time.The human visual system is good at quickly capturing interesting areas from massive data,and how to simulate the visual attention mechanism to accurately and efficiently extract important information and filter out redundant data has become a hot research topic.Visual saliency detection is one of the active fields in the community of computer vision,and useful to assist and refine a series of vision tasks.In recent years,more and more researchers have paid attention to it.Despite significant progress,visual saliency detection is still a challenging problem for some troublesome factors introduced by the limitations of single imaging system such as cluttered background and bad weather.Integrating multiple different but complementary cues,such as visible and thermal infrared spectral information(RGB-T),might be an effective way to increase the significant improvement of the detecting results.Therefore,we investigate several solutions by establishing the graph-based manifold ranking algorithm and fusing multiple spectral data in this dissertation.In particular,we propose a graph-based multi-task manifold ranking algorithm for RGB-T image saliency detection,and a spatiotemporal consistent ranking algorithm for RGB-T video saliency detection.To benefit from the complementary cues of multi-modal information,a novel multi-task manifold ranking algorithm is proposed for robust RGB-T image saliency detection.First,given an input image,we generate a set of superpixels using multi-modal information via the segmentation algorithm.Taking these superpixels as nodes,a graph is constructed with the spatial continuity and the appearance consistency.Second,we use the graph-based manifold ranking model for each modality,and introduce a modality weight for each modality and integrate into the ranking model to achieve adaptive fusion of different source data.Third,to perform collaborative fusion,we propose a cross-modal consistency constraint into the ranking model to form a joint model.Fourth,we develop an efficient optimization algorithm to the proposed model.Finally,in order to facilitate the evaluation of different saliency detection algorithms,we create a RGB-T dataset for image saliency detection.Our experiments on the newly established dataset demonstrate the presented method can substantially improve the accuracy of image saliency detection.Considering that the visible light video is vulnerable to weather,light and complex background,we propose a multi-modal video saliency detection algorithm.By introducing thermal infrared video information,we provide a collaborative manifold ranking model with the spatiotemporal saliency consistency for video saliency detection.Firstly,this dissertation makes full use of multi-modal information of each frame and modality weight variables to capture the video-based salient areas.Besides,we restrict the saliency consistency among different modalities.Secondly,a "stream processing" way is proposed to quickly batch detect the salient areas in consecutive video frames.To find out the spatiotemporal correlation in multi-modal videos,we make several reasonable assumptions and incorporate the spatiotemporal consistent constraint for unit processing consecutive frames.Finally,we establish a unified multi-modal video saliency detection platform,including video dataset,baseline methods and evaluation metrics.On the platform,we conduct extensive comparison experiments.The robust results demonstrate the rationality and effectiveness of the proposed model.