Multi-domain Image Sentiment Analysis Based On Convolutional Neural Networks

With the advent of the social era,it becomes commonplace for people to share their images and interact with others using comments on the website or apps(e.g.Flickr and MicroBlog).Identifying the sentiments precisely conveyed by the vast volume of available images has been proven highly useful for a variety of applications such as computer vision and image aesthetic analysis.Therefore,image sentiment analysis has gradually become a hot topic,especially the multi-domain image sentiment analysis which is full of challenges and mysteries.For example,images from different domains may share similar visual features while conveying opposite sentiments.In addition,texts and images belong to two different types of data,which are quite different and make it difficult to establish a direct connection between them.This paper aims to build a general model for classifying the sentiments of images,focusing on the topic of multi-domain image sentiment analysis.Most existing methods for image sentiment analysis are defective when they are used for multi-domain image sentiment analysis.There are still some unresolved challenges.For example,first,there are multiple types of data on the Internet,but the inherent relationship between different modalities of data is subtle and obscure which makes it difficult to discover and utilize.Second,there are so many domains about images on the Internet that it is unrealistic to build domain-specific classifiers for each domain.Third,images from different domains vary greatly,which is the reason that a classifier trained on images from one domain may perform badly on images from any other domain.At last,it's difficult to establish a direct connection between low-level visual features and high-level sentiments,since there is a huge gap known as "semantic gap".Therefore,there is still a lot of work to do for multi-domain image sentiment analysis.Therefore,we propose a novel hybrid unified model for multi-domain image sentiment analysis(GMCIS).It has two modules:a text category classifier using Long-Short Term Memory network(LSTM)and an image sentiment classifier using Convolutional Neural Network(CNN).The image sentiment classifier consists of several sub-components in which its number is equal with the number of different domains.Each sub-component is a precise classifier trained with a few images from the same domain,aiming to capture the specific visual features for this domain.The text classifier is served as a weight controller to set different weights of the sub-components,making it possible to capture the common features for multi-domains.GMCIS model combines the two modules by weighted training to predict the sentiment labels of images.Experiments show that our model achieves state-of-the-art performance,compared with some baselines.The main contributions of this paper are summarized as follow.1)It takes advantages of the images inherent visual features and textual comments to solve this problem,enriching the dimensionality of features.2)It connects images and texts through their contents,which narrows down the semantic gap between vision and language.3)It applies a variety of different strategies to extract textual features and examines their performance in the experiments,which obtains a comprehensive understanding of the textual comments and improves the performance of the proposed model as well.