Multimodal Sentiment Analysis Based On Facial Expression,Speech And Text

In recent years,with the rapid development of multimedia social platform,compared with the early research on sentiment analysis based on text data,more researchers show solicitude for sentiment analysis based on video,audio and text data.Multi-modal sentiment analysis can overcome the shortcomings of high error and low robustness of single-modal sentiment analysis by introducing information of multiple modalities.Moreover,it can make use of the complementary characteristics of different modalities to improve the representation ability of sentiment characteristics,so as to improve the ability of sentiment analysis.Human beings often express their sentiment in a variety of ways.Among them,expression,speech and text are the most common expressions.Under the background of multi-modal data development of multimedia social platform,this paper studies multi-modal sentiment analysis based on facial expression,speech and text,and carries out multi-modal sentiment classification for sentiment feature extraction and multi-modal feature fusion.The main contents of this paper are as follows:(1)In order to solve the problem that the representation ability of single-modal features is insufficient in the past,this paper improves three modal sentiment feature extraction methods: facial expression,speech and text.For expression modal,the pre-trained model and face key point features are used to obtain expression sentiment features;For speech modal,the spectrogram and convolution neural network are used to obtain speech sentiment features,and represent speech from two dimensions of time domain and frequency domain;For the text modal,the pre-trained word vector and sentence vector model are used to obtain the text sentiment features,and the pre-trained word vector and sentence vector model can well represent the text sentiment features through large-scale data training.(2)In order to solve the problems of high vector dimension and ignoring the correlation and difference between modalities in feature level fusion,an improved method of sentiment analysis based on attention mechanism is proposed to obtain multi-modal semantic information.This method adopts attention mechanism to learn the importance of each single-modal feature to its own influence,and the importance between different modalities,the multi-modal sentiment features including semantic information are obtained for sentiment analysis.The accuracy of multi-modal sentiment classification using attention mechanism reaches 82.71%.Compared with direct cascade fusion,the accuracy of sentiment analysis is improved by about 1%,and the ability of sentiment analysis is improved.(3)In order to solve the problem that most researches ignore multi-modal context,an improved method of sentiment analysis based on graph neural network is proposed to obtain multi-modal context.By constructing the context graph structure of multi-modal data,context is mapped into the multi-modal sentiment features for sentiment analysis with the use of graph neural network.The accuracy of multi-modal sentiment classification reaches 83.28%.The results show that the context information introduced by the graph convolution neural network makes the multi-modal sentiment features richer,and the robustness of the sentiment analysis model is stronger,which can further improve the ability of sentiment analysis.