Multimodal Sentiment Analysis Based On Deep Learning

With the development of the network,there are a large amount of data such as text,image and video.Sentiment analysis for such data is one of the important tasks in the field of artificial intelligence,and it can be applied for product recommendation,public opinion supervision or other aspects.This paper concerns on text sentiment analysis and sentiment-oriented cross-modal retrieval.For text sentiment analysis,first of all,we summarize and analyze the existing methods for text sentiment analysis and propose multi-task convolutional neural network for predicting text emotion distribution to overcome the disadvantages of these methods.We observe that only the single emotion label problem is concerned in the previous approaches,but the emotion ambiguity is ignored.To solve this problem,this paper introduces label distribution learning to model the emotions in the text,and combines it with the deep learning framework to predict the emotion distribution.Specifically,this paper proposes an end-to-end multi-task learning framework combining the classification loss function(Cross-entropy Loss)with the KL loss function(Kullback-Leibler Loss).During the training process,several tasks can be boosted.At the same time,this paper proposes a lexicon-based conversion strategy to generate weak emotion distribution from single label,and then feed into the proposed framework and the classification performance can be improved.In the experimental part,this paper conducts a detailed comparison and analysis with the existing methods.The results show that our method has achieved a better performance than the existing approaches on SemEval-2007 Task 14 for the label distribution prediction task.For classification task,the proposed method performs better on ISEAR,Fairy Tales,CBET,TEC,and SemEval-2018 Task 01 than methods for emotion classification.For the cross-modal retrieval task,firstly,we summarize the existing methods for cross-modal retrieval task and propose deep coordinated network to overcome the disadvantages of these methods.We observe that the previous methods only focus on the object level matching in the retrieval task,but ignore the high-level semantics of sentiment.To address this problem,we introduce the sentiment-oriented cross-modal retrieval problem and match different modality of data at both the object and sentiment levels to effectively promote the retrieval performance.Specifically,this paper designs a deep coordinated network,which adopts two branch networks to extract the respective feature representations of text and images,and then maps them to the public space by combining the classification loss with the sentiment constraints.In this network,the same modality data can be effectively distinguished and different modalities can be better matched and the performance of the retrieval task can be improved.The experimental results show that the proposed method has a better performance than the existing methods on the subset of VSO dataset.