Research On Sentiment Analysis Based On Multi-modal Information Fusion

With the dramatic advancement of video sites such as Tencent Video and iQiyi,more and more people share their opinions and comments through videos.These viewpoint vid-eos provide a large number of multi-modal samples.Analysis has therefore received un-precedented attention from academia and industry.Sentiment analysis is a basic research topic in the field of natural language.Traditional sentiment analysis tasks generally utilize only text modal,while multi-modal sentiment analysis uses multiple modal information such as text,audio,and vision.The variety of information categories brings more chal-lenges to sentiment analysis.This paper focuses on how to efficiently utilize multi-modal information.Research details are described in the following three aspects:Firstly,this paper proposes a multi-modal sentiment analysis method based on con-text-enhanced LSTM.This method not only captures multi-modal fusion information,but also utilizes context information to help sentiment analysis.Specifically,each modal is first encoded in combination with the context feature using LSTM which aims to capture the independent information within a single modality.Subsequently,merge the independent information of multi-modal,and utilize the other LSTM layer to obtain the interactive in-formation between different modalities to form a multimodal feature representation.Finally,the max-pooling strategy is used to reduce the dimension of the multi-modal representation,which will be fed to the sentiment classifier.Comparative experiments with the baseline methods show that the multi-modal sentiment analysis method based on context-enhanced LSTM proposed in this paper can effectively use context information to assist multi-modal sentiment analysis tasks,and the performance is greatly improved.Secondly,this paper proposes a multi-modal sentiment analysis method based on the hierarchical gating mechanism.Specifically,each modality is first encoded by Bi-LSTM which aims to capture the intra-modal interactions within a single modality.Subsequently,we merge the independent information of multi-modality using two gated layers.The first gate which is named as modality-gate will calculate the weight of each modal.And the other gate called temporal-gate will control each time-step contribution for final prediction.Finally,the max-pooling operation is used to reduce the dimension of the multimodal rep-resentation,which will be fed to the prediction layer.The experimental results show that the multi-modal sentiment analysis method based on the hierarchical gating mechanism has excellent performance and is significantly better than other baselines.Finally,this paper proposes a multi-modal sentiment analysis method based on modal fusion recurrent networks.Specifically,firstly,aiming at the problem that traditional methods cannot consider the influence of other modalities when modeling a single modal,which leads to the low ability to express single modal information,the Long-short Term Fusion Memory(LSTFM)is proposed.The advantage of LSTFM is that it can consider the information of other modalities while modeling a single modal,which greatly improves the information expression ability of a single modal and enhances the interaction between modalities to a certain extent.Secondly,utilizing attention mechanism to further fuse dif-ferent modal information.Thirdly,using another recurrence component to establish a con-textual relationship for the fused multi-modal information.Finally,using the multi-modal fusion feature to perform sentiment prediction.Experimental results show that the mul-ti-modal sentiment analysis method based on modal fusion recurrent network proposed in this paper can effectively capture multimodal sentiment information,has excellent perfor-mance on multi-modal sentiment analysis corpus,and is significantly better than other baselines.