Visual-Aspect Enhanced Sentence Network For Multimodal Sentiment Analysis

With the rapid development of the Web,multimodal data consisting of text and images in-crease rapidly in recent years,so multimodal sentiment analysis on these two modalities draws more and more attention from researchers.Most of the previous works concentrate on the fusion mechanism of the textual and visual modality in the text-image pair level.But documents with images are also becoming more and more popular in daily life,such as news,blogs and reviews.In these cases,there is a document with several images,different from the text-image pair,images are unaligned with text,and visual content is sparse compared with textual content for a document.So how to leverage both textual and visual content is a key problem for document-level multimodal sentiment analysis.This thesis proposed a novel neural network model using images to enhance the saliency of sentiment expressed by image-related sentences in a document through the gate and attention mechanism(called visual aspect gate unit in this thesis),instead of fusing the textual and visual features for the final sentiment classification.Experiments on two public multimodal datasets demonstrate the effectiveness of the proposed model whose performance is better than all the baseline methods.Ablation study and supplementary experiment show the contribution of each mod-ule of the model and effectiveness of the proposed visual aspect gate.And a visualization analysis intuitively shows how the model work to improve the performance.