Composition Analysis And Aesthetic Assessment For Natural Images

Computational image aesthetic prediction aims to endow the computers with the capability of perceiving aesthetics,analyzing the aesthetic attributes such as image composition,light,colors,depth,and automatically assessing the aesthetic quality just as human beings.Computational image aesthetic involves with different subjects,including computer vision,photography,painting,visual arts,brain sciences,aesthetic psychology.It is a new interdisciplinary advanced topic.The results of aesthetic analysis can be utilized as the criteria for parameter optimization of image enhancement and image inpainting algorithms.It can also be used to monitor the photographic system and evaluate the performance of style transfer algorithms.Additionally,image aesthetic analysis also shows great potential in many practical applications,such as artistic creation,robotics and electronic business.This dissertation presents a systematical study of image aesthetic analysis,which aims at improving the image composition modeling ability and aesthetic prediction.We focus on two critical issues,i.e.the image composition modeling and image aesthetic assessment.In the image composition modeling problem,we focus on the perspective modeling and vanishing point detection.In the aesthetic assessment task,this thesis explores image aesthetic prediction methods from single modal information to multimodal information.The main contributions are summarized as follows:1.A deep convolutional neural network-based perspective modeling method is proposed.Existing works for perspective modeling often need the prior knowledge of the vanishing point location.Besides,they only perform well in simple scenes.In contrast,we transform the perspective modeling problem into classification task,and use the deep convolutional networks to predict it.With its powerful feature representation capabilities,the proposed method not only can handle various complex scenes,but also do not need prior knowledge of vanishing point location.Finally,we also demonstrate the application of the perspective effects modeling in image composition analysis through an image retrieval system.By retrieving images that have similar contents with similar view points,the proposed method can provide on-site guidance to amateur photographers.2.A semantic-texture fusion network is proposed to detect the vanishing point which is one of the most important aesthetic attributes in image composition analysis.Traditional vanishing point detection methods mainly focus on Manhattan or man-made environments,which consists of a large number of line segments aligned to multiple dominant directions.However,in natural landscape scenes which is a significant genre in photography,the number of leading lines converging to the dominant VP may be small.Additionally,most landscape images often use one single dominant vanishing point to emphasis the overall composition.As a result,traditional vanishing point detection methods are difficult to apply in natural scenes.In this paper,we try to use the deep learning method to learn the textural features in the images.And then we tackle the vanishing point detection problem by fusing the textural and semantic features.The experimental results demonstrate that the proposed method can effectively find the dominant vanishing point in complex natural scenes and improve the detection results.3.A gated peripheral-foveal convolutional neural network based image aesthetic prediction approach is proposed.Learning fine-grained details is quite important in image aesthetic prediction task.Existing methods encode the fine-grained details by random cropping approach.However,the random cropping strategies may destroy the image content and undermine the semantic integrity.To this end,a novel Gated Peripheral-Foveal Convolutional Neural Network(GPF-CNN)is proposed which can mimic the functions of peripheral vision and fovea vision to extract the fine-grained details.Considering that the roles of peripheral vision and foveal vision are different in processing different scenes,we propose a Gated Information Fusion(GIF)network to adaptively weight these two branches.The experimental results demonstrate the effectiveness of the proposed method.4.A multimodal recurrent attention convolutional neural network for image aesthetic prediction task is proposed.Previous studies primarily depend on information from one modality and ignore information from other modalities.Besides,they neglect the selective attention mechanism when extracting visual features from images.In order to overcome these challenges,we propose the Multimodal Recurrent Attention Convolutional Neural Network(MRACNN)to mimic the selective attention mechanism in human vision system.Meanwhile,a Text-CNN is utilized to encode the high-level semantics from user comments.Extensive experiments demonstrate the proposed method can achieve superior performance.5.A multimodal self-and-collaborative attention network based image aesthetic prediction approach is proposed.Traditional multimodal approaches usually face two challenges.First,these methods usually extract visual features via the convolution.However,the convolution only processes the local neighborhood in space and is difficult to encode the long range interactions between different image regions.Besides,there is a natural symmetry between the images and user comments.Existing methods ignore the inter-relationships between the two multimodal features.In order to address the above problems,we propose a Multimodal Self-and-Collaborative Attention Network(MSCAN).In MSCAN,the self-attention mechanism is used to encode the scene context of the input image.The co-attention module is used to encode the mapping relationships between the multimodal features.Extensive experiments demonstrate the effectiveness of the proposed MSCAN on three aesthetic prediction tasks,such as the aesthetic score distribution prediction,aesthetic quality classification and aesthetic score regression.