Quantum-like Interactive Models For Multimodal Sentiment Analysis

With the rapid development of social networks,more and more users are used to sharing their opinions or communicating with others through various media forms such as texts,images,videos,etc.Mining and capturing the subjective information of multimodal documents is of great significance for public opinion analysis,marketing and investment forecasting.Hence,multimodal sentiment analysis has become one of the core research subjects,and has attracted an increasing attention from both academia and industry.However,the research community's understanding of interaction is not deep enough and unified.The traditional methods that are based on classical probability theory are insufficient in dealing with interaction modelling.How to accurately and comprehensively model complicated interactions is the key problem that plagues the field.To address this issue,this paper conducts a study on three typical interaction problems in multimodal sentiment analysis tasks,establishes an integrated theoretical system of multimodal quantum-like interaction modelling and develops the corresponding multimodal quantum-like interaction models.The major innovations of the work presented in this paper are summarized as follows.First,aiming to solve the problem of the interactions between terms,e.g.,the interactions among textual words and the interactions among visual pixels,this paper proposes a quantum-like semantic representation model and a quantum-like sentiment representation model.For text,all words can be seen as quantum events,which are represented by any orthogonal projectors.Compound words can be seen as the superposition events.After defining projectors for each textual word and compound word,this paper can represent a document with a sequence of projectors temporarily,and thus use the Maximum Likelihood Estimation(MLE)to train density matrices.For an image,this paper considers it as a document of visual words,in which each visual word is equivalent to a word in document.Therefore,this paper can train the density matrix to represent the image.Theoretically,compared with vector-based representation,density matrices can better encode the semantic dependencies and their probabilistic distribution information.Second,aiming to solve the problem of the interactions between modalities,e.g.,the interference between textual predictions and visual predictions,this paper proposes a quantum interference inspired multimodal decision fusion approach.Multimodal sentiment analysis involves a complex decision process,in which different modalities often intertwine together to express a common sentiment polarity.The sentiment information of different modalities will influence the final decision simultaneously,leading to the interference-like phenomenon.Hence,this paper draws an analogy to the double-slit experiment in multimodal sentiment analysis,and uses the wave function to formalize the analogy.This decision fusion approach could model the correlation between modalities.Third,aiming to solve the problem of the contextual interactions between textual utterances,e.g.,how one speaker influences another,this paper first creates a manually labelled conversational dataset for advancing interactive sentiment analysis models.Then,this paper presents an operational definition of interaction in conversational text sentiment analysis task,and studies the multidimensional nature of contextual interaction,which includes understandability,credibility,and influence.In order to explicitly model interactions between speakers,this paper proposes an interactive Long Short-Term Memory(LSTM)network.Last,this paper develops the quantum-like contextual sentiment analysis networks(QCN)model.Specifically,a density matrix based convolutional neural network(DM-CNN)is proposed to capture the interactions within each utterance(i.e.,the correlations between words),and a strong-weak influence model inspired by quantum measurement theory is developed to learn the interactions between adjacent utterances.Extensive experiments are conducted on the MELD and IEMOCAP datasets.The experimental results demonstrate the effectiveness of the QCN model.Fourth,aiming to solve the problem of different types of interaction,this paper combines the aforementioned three methods,establishes an integrated theoretical system of multimodal quantum-like interaction modelling and develops the quantum-like multimodal interactive networks(QMN)framework.Specifically,QMN consists a quantumlike semantic representation model for extracting textual and visual features,a multimodal decision fusion approach inspired by quantum interference theory(QIMF)to capture the interactions between different modalities,and a strong-weak influence model inspired by quantum measurement theory to model the interactions between adjacent utterances.To sum up,our research work is helpful for deepening the understanding of complex interactions,brings fresh air for developing interactive sentiment models,and has important scientific and social values.