Visual Sentiment Analysis With Visual Attribute Detection

Nowadays,with the development of the Internet technology and the rise of the social media platforms,massive users upload multimedia content such as text and pictures to express their attitudes and opinions at all times,which makes the Internet a rich sentiment resource.In the meantime,as an important research area of user behavior analysis,image sentiment analysis has attracted increasing attention.Most of the existing works try to discover the emotional information from the global perspective of the whole picture.The focus of these works is how to build a more robust neural network.However,few attentions have been paid to the role of local regions towards image sentiment analysis.In addition,the semantic gap between low-level visual features and high-level semantic features also needs to be solved in image sentiment analysis.This paper proposes a sentiment prediction algorithm based on visual semantic embedding and attention mechanism.Firstly,to solve the semantic gap between visual features and emotional semantics,a visual semantic embedding module which is based on an auto-encoder,is designed.The hidden layer is trained to reconstruct visual features and simultaneously regress to attribute.After training,we obtain a visual semantic features which can learn the compact of the visual features and semantic features.Secondly,we design a sentiment prediction module with attention mechanism to highlight the role of local regions.The module extracts a set of salient regions in the image through the salient regions detection network and leverages the attention mechanism to establish the association between salient region features and visual semantic embedding features.After calculating the attention scores of salient region features and pooling these features under the guidance of the attention map,we build a sentiment classifier to predict image sentiment.The algorithm integrates auto-encoder,attention mechanism and salient region detection network into a common model effectively,pays attention to the role of local salient areas in the image,and utilizes the auto-encoder to learn the latent features of visual features and attribute features,which bridges the gap between image features and semantic features to some extent.In this project,we set three comparative experiments on the VRD benchmark dataset to verify the effectiveness of each module in our algorithm.By comparing with five state-of-the-art methods,the accuracy of the proposed method is superior to those of the state-of-the-art algorithms,which shows that our method significantly improves the performance of the image sentiment prediction on the VRD dataset.