The Research Of Cross-modal Sentiment Analysis For Images On Social Media

Sentiment analysis,also known as opinion mining,is a fundamental research domain about the opinions,emotions,affections,and attitudes expressed by individuals about many topics.To understand the connections between users' emotion and their behavior,researchers have made efforts on sentiment analysis.Sentiment analysis first became the most active research field in natural language processing.Later,workers carried out extensive researches on other sentimental areas such as data mining,image recognition,text mining,and information retrieval.This is because sentiment is the central motivation of all human activities and the critical factor that affects people's behavior.The dependence and needs of users to predict human activities make sentiment analysis on social media have great practical significance.However,multi-modal data with many texts and images is gradually replacing the single-modal corpus as the primary medium of sentimental transmission on social media.Then the multi-modal image sentiment analysis has drawn increasing attention.The image sentiment analysis on social media websites mainly faces two challenges: one challenge is the semantic gap of an image and sentiment.The sentiment is subjective,and the information contained in the image is clear and objective.How personal emotions are represented with accurate data.The other challenge is the fuzziness and complexity of an image,the sentiment expressed by an image with the same contents may be different under different semantic background.In this thesis,images and texts on social media are used as research objects.Visual contents and textual semantic are learned to express potential sentimental semantic features.A multi-layer network for alignment is constructed to bridge the cross-modal correlations of an image and its corresponding caption.Besides,an attention-based component is improved to explore the structured semantic dependencies between entities in the image captions.For the sentimental perdition,at last,the sentimental graph-based named entity disambiguation method is discussed and evaluated.The main research contents and contributions of this thesis are as follows:1)Named entity disambiguation based on semantic similarityTo obtain the sentimental polarity and intensity of words in the short text on social media.In this thesis,a named entity disambiguation method based on semantic similarity is proposed,which combines contextual information of entities in short text with attributes of entities in knowledge graph to calculate semantic similarity.In addition,this thesis proposes a joint learning model based on word embedding and category embedding.This model calculates the similarity between words and their categories to improve the accuracy of entity disambiguation.The comparative experimental results verify the effectiveness of the semantic similarity calculation method proposed in this thesis,which can effectively obtain the sentimental polarity and strength of entities in short texts on social media.2)Inner Relation modeling in Aspect-based Sentiment AnalysisAspect-based sentiment analysis predicts the sentiment polarity of a specific target word in a given text by analyzing its dependencies.Research shows that using the attention mechanism to model the target word and its context can help learn more effective representations for sentiment classification.To analyze the hierarchical relationships contained in complex texts,in this thesis,an aspect-based sentiment analysis method with the attention mechanism is proposed to model inline relations of words in the text.Specifically,a recurrent neural network is designed to analyze the dependence of the target word and its context in a given text to predict the emotional polarity.The model proposed is evaluated on public data sets,and the experimental results verify the effectiveness of the method proposed.Moreover,the case study of visualization shows that the sentiment prediction method is interpretable and stable.3)Cross-modal image sentiment analysis based on text semantic correlationTo realize image sentiment classification by using complementary multi-modal information,the relationship between an image and its captions on social media is investigated,and an end-to-end method for image cross-modal sentiment analysis is proposed in this thesis.This method integrates the work of two former parts and maps the image contents to the corresponding words for analyzing the dependency of the words in the text with the query mechanism.Finally,the image sentiment polarity is predicted.In addition,based on the visual contents and textual semantics,this thesis proposes a joint attention network to bridge the correlation of an image and its caption.The experimental results show that the end-to-end cross-modal image sentiment analysis method proposed in this thesis is effective.Ablation studies verify each component in the model is useful.Ablation experiments and case studies show the effectiveness of the joint attention network proposed for associating image contents with text semantics.In this thesis,an end-to-end image sentiment analysis framework is proposed,and three key components in the framework are studied.The main research work of this thesis focuses on the cross-modal image sentiment analysis on social media,which supplements domestic and foreign research in this field.This research provides an effective solution for the lack of interpretability and relevance on image sentiment analysis.Moreover,the method and its realization offer a new idea and approach for the research and application of multi-modal data fusion.