Research On Visual Saliency Detection With Comprehensive Information

The Human visual system works as a filter to allocate more attention to the attractive and interesting objects for further processing.Visual saliency detection model simulates this system to perceive the scene,and has been widely used in many vision tasks,such as segmentation,retrieval,compression,coding,quality assessment,and so on.With the development of acquisition technology,more comprehensive information,such as depth cue,inter-image correspondence,or spatiotemporal relationship,is available to extend single image saliency detection to RGBD saliency detection,cosaliency detection,or video saliency detection.RGBD saliency detection model focuses on extracting the salient objects from RGBD images by combining the color and depth information.Co-saliency detection model introduces the inter-image correspondence constraint to discover the common salient objects in an image group.The goal of video saliency detection is to locate the motion-related salient object in video sequences,which considers the motion cue and spatiotemporal constraint jointly.In this thesis,comprehensive information is explicitly explore to address the challenges in different saliency detection tasks.Specially,the main works of this thesis are summarized as follows:(1)For the stereoscopic images,considering the quality of the depth map and multiple cues fusion,a novel saliency detection method is proposed.First,according to the observation of depth distribution,a confidence measure for depth map is designed to reduce the negative influence of poor depth map on saliency detection.Moreover,a novel stereoscopic compactness saliency model is defined by integrating the color and depth information.In addition,a depth-refined foreground seeds selection mechanism is presented to assist in foreground saliency calculation by integrating color,depth,and texture cues.At last,the complementary compactness saliency and foreground saliency are fused to generate the final saliency map.(2)In order to fully exploit the depth and inter-image correspondences,this thesis first attempts to address the co-saliency detection from an RGBD image group,in which the depth information is introduced as a novel cue in the designed model.In order to explore the inter-image relationship,the similarity matching methods on two levels are proposed.The first one is the superpixel-level similarity matching scheme,which focuses on determining the matching superpixel set for the current superpixel based on three constraints from other images.The second is the image-level similarity measurement,which provides a global relationship on the whole image scale and works as a weighted coefficient for inter saliency calculation.Finally,the cross label propagation method is proposed to optimize the intra and inter saliency maps in a cross way,and generate the final co-saliency map.(3)The existing co-saliency detection methods mainly rely on the designed cues or initialization,and lack the refinement-cycle.Thus,an effective co-saliency framework for RGBD images based on the refinement-cycle model is proposed,which integrates the addition scheme,deletion scheme,and iteration scheme.The addition scheme is used to enrich the saliency regions through the depth propagation and saliency propagation.Note that,a novel depth descriptor,named depth shape prior,is proposed in depth propagation to introduce the depth information and enhance the identification of co-salient objects.In the deletion scheme,the inter saliency is formalized as a common probability function to capture the inter-image correspondence.The iterative optimization scheme is designed to achieve more superior co-saliency result in a cycle way.The proposed method effectively exploits any existing 2D saliency model to work well in RGBD co-saliency scenarios.(4)In order to balance the effectiveness and efficiency for inter-image correspondence capturing in co-saliency detection,a novel RGBD co-saliency model is proposed based on hierarchical sparsity reconstruction and energy function refinement.The multi-image correspondence is formulated as a hierarchical sparsity reconstruction framework,where the global sparsity reconstruction captures the global characteristic among the whole image group through a common foreground dictionary,and the pairwise sparsity reconstruction model utilizes a set of foreground dictionaries produced by other images to explore local inter-image information.Finally,in order to improve the intra-image smoothness and inter-image consistency,an energy function refinement model is proposed,which includes the unary data term,spatial smooth term,and holistic consistency term.(5)Combining the spatial saliency in the single frame,the temporal cue in the inter frames,and the global constraints among the whole video,a novel method to detect the salient objects in video is proposed based on sparse reconstruction and propagation.The single-frame saliency is calculated to represent the spatial saliency in each individual frame via the sparsity-based reconstruction,where the motion priors are defined as the motion compactness and uniqueness cues.Then,an efficient sparsitybased saliency propagation is presented to capture the correspondence in the temporal space and produce the inter-frame saliency map.Specifically,the salient object is sequentially reconstructed by the forward and backward dictionaries.Finally,in order to attain the spatiotemporal smoothness and global consistency of the salient object in the whole video,a global optimization model is formulated,which integrates unary data term,spatiotemporal smooth term,spatial incompatibility term,and global consistency term.