Deep Learning Framework Based 3D Image Visual Saliency Detection

Driven by the development of emerging 3D multimedia technologies,3D image visual saliency has become a well-known challenge in visual saliency.Different with 2D image,the application of 3D image or video can provide users with depth perception and immersive visual experience with extensive market demand and application value.Extra depth information need to be considered in 3D image visual saliency detection.However,the traditional handcrafted feature extraction method cannot extract the high-level semantic features of depth information.Thus,in this dissertation,we propose deep learning based 3D image visual saliency detection algorithm.The main work of this dissertation includes:(1)Due to most of the existing 3D image visual saliency detection models adopt the handcrafted feature extraction method which can only be used to extract low-level features of image and cannot be used to extract high-level semantic features of image.Therefore,we propose a deep learning features inspired 3D image visual saliency detection model.The pretrained Convolutional Neural Network(CNN)model is employed to extract the feature vectors for multi-level image regions.Then,a Neural Network(NN)based saliency prediction network is applied to inference the saliency value of the local region from the multi-level feature vectors.Finally,the color and depth saliency map are fused to obtain the final saliency map for 3D image.This algorithm solves the deviation problem that the existing 3D image visual saliency models adopt the handcrafted feature extraction method to a certain extent,and improves the performance of the proposed model.(2)The multi-scale image segmentation used in the algorithm proposed in the research work(1)is redundant and the pre-trained CNN model is initially applied to image classification task rather than image visual saliency detection task.According to the shortages of the algorithm proposed in the research work(1),we propose deep neural networks(DNNs)based framework towards effective saliency detection in 3D images.The framework of the proposed algorithm consists of multi-resolution region wise saliency detection network(MCRWP-Net)and pixel wise spatial fusion network(PWSF-Net).This algorithm solves redundancy problem of multi-level image segmentation method by using super-pixel based image segmentation method.A deep neural network is employed to train a specific network model for 3D image visual saliency detection task.The performance of the proposed model has been improved.(3)The algorithms proposed in the research works(1)and(2)both predict saliency values for regions.However,sharing the same saliency value in a whole region will increase the prediction deviation.Hence,we propose multi-channel fully convolutional network based 3D image visual saliency detection model.The framework of the proposed algorithm consists of multi-channel fully convolutional visual saliency prediction networks(PredNets),learning based center bias priors(Center Bias Priors)and an inter channel fusion of dense saliency prediction net.The pixel wise PredNets construct prediction networks that have three-channels(color,depth and joint channels of color and depth).Then,an inter channel fusion of dense saliency prediction model is used to learn the spatial correlation and difference between color and depth information.Finally,the final 3D image visual saliency map is obtained.The performance of the proposed model has been improved.