Multimedia Emotion Analysis Enhanced By Emotion Relationship Patterns And Attention Detection

The progress of science and technology has led to the vigorous development of multimedia technology.Massive multimedia data are produced every moment.These multimedia data carrying human emotion is a huge treasure,which are worthy of our exploration and mining.In recent years,emotion analysis technology has received more and more attention.Efficient emotion analysis algorithm can help people understand and use multimedia data,reduce production and management costs,and improve the efficiency of human-computer interaction.This thesis focuses on two problems related to emotion analysis tasks:multimedia emotion tagging and image emotion distribution learning.One of the difficulties of emotion tagging task is how to model the emotion relationship pattern,while the key of image emotion distribution learning is how to extract the emotion features with representation ability.For the former,we leverage the graph convolutional network and adversarial learning strategy to capture the emotion relationship patterns at the feature level and the label level of the model,respectively,and further explore the association between multi-dimensional emotions.For the latter,we introduce an attention mechanism to enhance the ability of the algorithm to perceive emotional content in the image from both global and local perspectives.In general,this thesis proposes an emotion analysis method enhanced by emotion relationship patterns and attention detection.The details are as follows:1.This thesis proposes a multimedia emotion tagging algorithm based on emotion relationship patterns.In this method,local and global emotion relationship pat-terns are applied at the feature level and label level of the model,respectively,to assist the algorithm in further mining the emotional content contained in mul-timedia data.At the feature level,we build the emotion relationship graph to model the local relationship among multi-dimensional emotions,and generate the emotional representation of each emotion through the graph convolution op-eration.These emotional representations are injected into the model as external knowledge.At the label level,adversarial learning is leveraged to constrain the generated multi-dimensional emotional labels.An additional discriminator is in-troduced to gamble with the model to ensure that the labels obtained are as close to the real labels as possible.In the experimental stage,we conducted comparative experiments on four multimedia emotion databases,namely Music,NVIE,Film-stim and Memorability,respectively,to prove the performance of the algorithm compared with related works.The effectiveness of different modules of the algo-rithm and the influence of different experimental parameters on the performance of the model are illustrated by ablation experiments.2.This thesis proposes an image emotion distribution algorithm based on attention detection and correlation modeling.Previous works often abstract visual repre-sentations from the global perspective of images,thus neglecting the emotional content contained in local regions.To address this problem,we propose a class-wise emotion detection module,which generates the corresponding attention map for each emotion through multi-modal bilinear pooling.These attention maps show how likely each regions of the image is to elicit the corresponding emotion.On this basis,global emotion attention map can be generated by the proposed method,which enables the model to explore the emotional regions from both global and local perspectives.In addition,graph convolutional network is in-troduced to enhance the communication between the emotional regions and help the model extract more comprehensive emotional representations.In the exper-imental stage,we compare the algorithm with related works on the FlickrLDL database and the TwitterLDL database,and the experimental results prove the superiority of the proposed method.Visualization experiments further demon-strate the effectiveness of the attention map generation mechanism proposed by the algorithm,and ablation experiments prove the function of each module of the proposed method.To sum up,focusing on the thesis of emotional content analysis of multimedia data,we propose to construct emotion relationship patterns at the feature level and label level to improve the difficulty of emotional label modeling.Meanwhile,we introduce local and global emotional attention maps to strengthen the representation ability of image emotional features and further improve the performance of emotion analysis.