Research And Application On Multimodal Sentiment Analysis Based On Combination Of Image And Text

With the development of information and communication technology,social media is integrated into the daily life of the general public.Users post their opinions on social platforms and their subjective emotional messages spread and ferment in the virtual network,which in turn affects the processes and developments in the real world.Nowadays,the content posted by users is no longer limited to a single text message,but rather a variety of multimodal data messages,including text,images and videos.However,with the emergence of multimodal social data,sentiment analysis has become a difficult problem across multiple domains and disciplines.When dealing with multimodal data,it is not only necessary to address the heterogeneity of multimodal data,but also to take into account the fusion and classification of different sentiments expressed by different multimodal data.At the same time,sentiment analysis in different scenarios is influenced by a variety of factors,including data type and cultural context.Therefore,scenariospecific knowledge is needed to guide the design of sentiment analysis applications.The main innovation and work in this paper will focus on addressing the above issues..(1)The problem of heterogeneity between image and text data is addressed.In order to better fuse the features of text and images,this paper uses a bilinear feature fusion algorithm to fuse the features of text and images.At the same time,an attention mechanism is added to the bidirectional long and short-term memory network for extracting text features,and a bilinear convolutional network is used for extracting image features,and a multimodal sentiment analysis model based on bilinear convolution and bilinear feature fusion(BCNN-BFF).The experimental results show that the proposed model can effectively solve the problem of fusion of graphical features,and the accuracy rate of sentiment analysis is 76.89%,which is an increase of 2.83% compared to the FENet multimodal sentiment analysis model.(2)To address the problem of inconsistent sentiment expressed between modalities,the widespread phenomenon of sarcasm(irony and satire)in cross-modal sentiment analysis,this paper proposes a multimodal sarcasm detection model(MSDM)based on the combination of text and image.In order to accurately identify the sarcasm expressed in the different multimodal data,an attention mechanism is first added to the feature extraction algorithm from both text and image modalities to reduce noise and redundant features.Then image attributes are extracted from image data as the third modality to increase the model’s understanding of the data and improve its generalization ability.Finally the proposed model is validated for performance on the Twitter sarcasm dataset.The experimental results show that the MSDM model has an accuracy of 88.99%,which is better than the BERT multimodal sarcasm detection model.(3)Designing a prototype system for multimodal sentiment analysis of hotel reviews.This paper designs and develops a prototype system for multimodal sentiment analysis of hotel reviews based on multimodal sentiment analysis and multimodal irony detection using algorithms.The graphical data of customer reviews is used as input,and the multimodal sentiment analysis system is used to detect sentiment polarity and to present the interface.A dataset of 3200 hotel reviews collected from Ctrip.com was used for detection and analysis,which was used to validate the performance of the system.