RGB-T Salient Object Detection Via Fusing Multi-level CNN Features

Salient object detection aims to identify the most visually distinctive objects or regions in an image,and has become one of the most active research directions in cognitive field and computer science in recent years.As a preprocessing step,salient object detection plays a critical role in many computer vision tasks,including visual tracking,image recognition,content-based image compression,image fusion and so on.So far,numerous salient object detection methods have been presented.However,most of these methods are designed for the RGB images only,which may fail to discriminate salient objects from backgrounds when being exposed to challenging conditions,such as poor illumination,complex background,and low contrast.Instead of improving RGB-based saliency detection,which might be hopeless,this work explores an alternative way to take advantage of the complementary benefits of RGB and thermal infrared images.Specifically,a novel RGB-T salient object detection via fusing multi-level CNN features is proposed.Experimental results on several public RGB-T salient object detection datasets verify the validity of the proposed algorithm.In summary,the main contributions of this work are as follows:First,some salient object detection methods and computer vision algorithms based on RGBT images are summarized.Especially,three existing algorithms which are highly relevant to this dissertation are discussed in detail.These include deeply supervised salient object detection with shot connnections,multi-task manifold ranking with cross-modality consistency for RGB-T saliency detection,and learning multiscale deep features and SVM regressors for adaptive RGB-T saliency detection.Secondly,an end-to-end RGB-T salient object detection algorithm based on multi-level depth feature fusion is proposed to solve the following problems.The performance of RGBinduced salient object detection is still much room for improvement in challenging scenarios,and RGB-T salient object detection is difficult to fuse the complementary information of multi-modal image effectively,and deep learning based RGB-T detection algorithm can not achieve end-to-end detection yet.The main steps of this work are as follows:(1)Extract multi-level features of RGB or thermal infrared image from different depths of VGG-16 Net.(2)Multiple djacent-depth features combination modules are constructed to extract multilevel features of input images containing rich spatial details as well as semantic information.(3)To obtain cross-modal features,a multi-branch group fusion module is put in place to fuse the features of RGB-T image pairs at each level.(4)A joint attention guided bidirectional message passing module is presented to get multiple side outputs for different levels.(5)fuse side outputs further for final saliency map prediction.Finally,the proposed algorithm is implemented on the MATLAB R2014 b platform with the Caffe toolbox and a NVIDIA 1080 Ti GPU(with 11 G memory).The training and testing methods of the proposed RGB-T salient object detection algorithm for RGB-T images are elaborated in detail,and a comparative experiment is designed to compare this algorithm with the state-of-the-art methods on several publicly available datasets.Experimental results demonstrate the superiorities of the proposed algorithm over the state-of-the-art approaches,especially under challenging conditions,such as poor illumination,complex background and low contrast.