Research On Image Sentiment Analysis Based On Deep Learning

With the rapid development of mobile terminal technology and social media technology,there are huge amounts of multimedia content appears in social media every day.the most typical of which are images and videos.Users often want to share their experience or view of things through the images and videos.Researchers can dig deeper into users' habits and mental states by analyzing these users' data so that they can analyze the users' demands better and serve the users better,which can improve the users' experience.User sentiment analysis is an important part of user behavior analysis.In this paper,we take image sentiment as the research object,and thoroughly study the image sentiment classification and the image retrieval based on sentiment with deep learning.Classification problem is a typical problem in pattern recognition.The performance of classification often depends on two aspects:one is the selection of feature,and the other is the classifier selection.This thesis mainly studies the problem of feature selection in image sentiment analysis.Different from the traditional classification problems,image sentiment classification is a more abstract and advanced problem of image understanding,so that the feature selection is more challenging.Similarly,for the image retrieval based on sentiment,accurate image feature representations are the core to ensure the performance.Aiming at such problems,we propose a variety of different feature extraction approaches in this thesis.The contributions of this thesis can be summarized as follows.1)This thesis proposes an image sentiment classification approach based on deep semantic features.The traditional approaches always use low-level feature descriptors for object detection or object classification in image sentiment classification or features designed based on the characteristics of aesthetics and psychology.However,these features lack the overall perception of an image,so that the performance of image sentiment classification is not good enough.As we all know,objects and scenes are the core of an image.Features based on them are more advanced semantic features compared with features mentioned before,and they are more suitable and reasonable to represent images.Traditional approaches based on semantic features mainly have two problems.On the one hand,traditional semantic features are constructed from the low-level features designed by humans,and they are not accurate to represent the semantic contents in images;on the other hand,different semantic features are not taken into consideration at the same time.These two reasons restrict the performance of previous approaches.In this thesis,approaches based on deep semantic features are proposed,and better classification performances are achieved.Specifically,on the one hand,we propose and verify the approaches based on different deep semantic features with different levels of abstraction;on the other hand,we propose approaches based on improved multi-feature fusion strategies,which improve the performance of image sentiment classification in comparison with the ones with traditional multi-feature fusion strategies.2)This thesis proposes an image sentiment classification approach based on visual attention.The common approaches extract the image feature from the whole image regions without any distinction.This means they encode an entire image into a fixed dimensional representation,while leaving the regions of the image that are most indicative to infer the sentiment not fully exploited.To address this problem,we propose an image sentiment classification approach based on visual attention.Specifically,on the one hand,we directly use the result of saliency detection as the relative importance of different regions for image sentiment classification,and get the image feature by weighting different local region features;on the other hand,we train a network to generate the relative importance of different regions automatically in terms of local region features,and then get the image feature by weighting different local region features.These two approaches improve the performance of image sentiment classification by optimizing the mechanism of feature generation.Moreover,we also analyze the influence of different visual attention learning methods on the learnt visual attention model and the performance of image sentiment classification.3)This thesis proposes an approach of image feature extraction for image retrieval based on sentiment.Extracting discriminative features is the core of image retrieval based on sentiment.Traditional image retrieval approaches based on deep learning directly take the output of one layer in the pre-trained deep neural networks for object or scene classification.Although they improve the performance of image retrieval in comparison with the traditional approaches based on hand-crafted features,they ignore the two important concepts,which are small intra-class distance and large inter-class distance.In this thesis,we design new loss functions to address these in the training of the networks for feature extraction.The networks trained under the supervision of new loss functions can extract more discriminative features for image retrieval based on sentiment,and this boosts the performance.A lot of experiments have been done to verify the proposed approaches in the thesis.The experimental results show that the approaches based on deep semantic features can achieve better performance of image sentiment classification compared with the approaches based on traditional semantic features,and the approaches with our new multi-feature fusion strategies can further boost the performance.The approaches with visual attention can get better classification result by extracting better feature representations.Moreover,the new designed loss functions can supervise the networks for feature extraction better by addressing the importance of small intra-class distance and large inter-class distance,which makes the trained networks can get more discriminative features,and improves the performance of image retrieval based on sentiment.