Light-Field Saliency Detection Via Convolutional Neural Networks

Saliency detection has always been one of the most important tasks in computer vision,which is critical in some applications,such as visual tracking,image compression,and object recognition,etc.The existing saliency detection methods based on RGB images or RGB-D images are easily suffered from complex backgrounds,illumination,occlusion,and other factors,which leads to inferior detection performances.Therefore,there is a challenge to improve the robustness of saliency detection.In recent years,commercial and industrial light field cameras based on micro-lens arrays inserted between the main lens and the photo sensor have taken light field imaging into a new era.Researchers begin to solve saliency detection problem from a new perspective.The light field not only records the spatial information but also the directions of all incoming light rays.The spatial information and angular information inherent in a light field implicitly contains the geometry and reflection characteristics of the observed scene,which can provide reliable prior information for saliency detection such as background clues and depth information.At present,the saliency detection based on light fields has been widely concerned.Existing saliency detection approaches are fragile and the saliency object by these methods are incomplete when dealing with challenging scene.In this thesis,we make use of the view information in the light field and the powerful feature expression ability of the convolutional neural network to solve these challenging scenarios for saliency detection.The main contents are listed as follows.(1)We introduce the related work of saliency detection,analyze the research significance of this task,and discuss the basic theory of light field,acquisition methods of light field data,image representation and the concept of saliency detection.(2)We acquire the light field by Lytro Illum camera and built a rich and high-quality dataset.This dataset contains 640 indoor and outdoor scenes with complex backgrounds,similar foreground background colors,multiple light sources,multiple salient objects,etc.It provides a more powerful data support for further research in the field of light field saliency detection.(3)In order to further explore the positive effects of spatial characteristics and multiview characteristics of light field on saliency detection,we designed a saliency detection algorithm based on micro-lens images.In order to directly model angular changes at one pixel location,we designed three different Modal Angular Changes(MAC)modules,which capture the view dependence in a micro-lens image through non-overlapping convolution on each micro-lens image.This algorithm firstly uses the Modal Angular Changes modules to process the view information of the micro-lens image,and then inputs the obtained feature maps into the modified deeplab-v2 model to predict saliency maps.The experimental results show that the angular changes are consistent with the viewpoint variations of micro-lens images,and the effective angular changes of each pixel may increase depth selectivity and the ability for accurate saliency detection.Compared with the current state-of-the-arts methods,our approach can effectively improve the saliency detection performance of complex scenes.(4)Considering that the saliency detection algorithm based on micro-lens images has poor learning ability for color contrast information in the scene,we propose a light field saliency detection algorithm based on two-channel fusion,which consists of twostream network with a micro-lens image array and central a view image as inputs.In order to better explore the relationship between the two channels,a fusion module is introduced for feature learning.The experiment verified that the two-channel fusion network used the feature of the central view image channel to fine-tune the feature of the micro-lens image channel,and supplemented the learning ability of the image channel of the microlens to the scene color contrast information which makes the algorithm more robust to the saliency detection for complex scenes.