Research On Deep Learning Based Image Saliency Prediction

Saliency prediction is a subtask in the field of computer vision and it is closely related to human visual attention mechanism.Modeling of saliency prediction not only gives insight into human visual mechanism,but also shows much potential in image cropping,object objection,tracking,etc.With the wide application of deep learning in the field of saliency prediction,the performance of saliency prediction has been greatly improved.However,there are still a lot of problems to be solved.Datasets,metrics,models are three elements should be considered and updated to improve the performance of saliency prediction.This thesis aims to design more powerful deep learning based saliency prediction models.The focus lies on integrating action-aware features into saliency models and combining multi-layer features of neural network for saliency prediction.Deep learning based saliency models cannot predict well on image areas of highlevel semantics such as some faces,texts,objects of action,etc.This thesis focuses on objects of action which concerns salient objects interacted with by person or salient regions containing possible human action.A method of integrating action-aware features into saliency models is proposed to improve models' performance on objects of action.By weakly supervised learning,a classification network trained with image level labels can locate areas about labels on image.This thesis extracts action-aware features from a classification network trained on image based action datasets,then fuse action-aware features and contextual features from saliency models to predict saliency maps.The experiments on three benchmark datasets show that the proposed saliency model with the action-aware features achieves better performance.This study is an attempt on explicitly integrating objects of action into deep saliency models to improve performance and proposes a simple and extensible method by weakly supervised learning.A majority of deep learning based saliency models rely on high-layer features of neural network but ignore lower-layer features which also matters.In this work,in order to make full use of multi-layer features and get more robust prediction results,a novel model fusing multi-layer features of neural networks is proposed.According to the characteristics of saliency prediction problem,a method improving high-layer features' performance by lower-layer prediction is designed.Firstly,a top-down module is adopted to enrich low-layer semantics in order to get predictions from low layers.Then,a dilated convolution module is designed to extract more global context information to improve performance of all branches.Finally,high-layer features are fused with low-layer prediction to produce more powerful predictions.This thesis designs such a novel architecture called BML-Net for saliency prediction and BML-Net shows great performance on two benchmark datasets.