Sentiment Recognition For Short Annotated GIFs Using Visual-Textual Fusion

With the rapid development of social media,the increasing demand for short videos is strong and sustained,while video sentiment analysis has become a hot and important topic in video understanding researches.In this work,we propose an effective multi-model sentiment recognition approach using visual and textual fusion for the short annotated video such as GIF video.Firstly,3-D convolutional neural network(3D CNN)and convolutional long short term memory(ConvLSTM)techniques applied for the given short GIF video can be organically built in our proposed recognition system to visually perceive the assumed sequence feature.Then,the output probabilities of softmax classifier for three defined categories can be calculated and handled as the visual sentiment score for the given short videos.Next,for the descriptive and textual sentences with respect to the given short annotated videos,we exploit the Synset forest to extract the sets of the meaningful sentiment words from the given sentences and utilize the SentiWordNet3.0 model to evaluate them to acquire the textual sentiment score.Then,we design a joint visual-textual sentiment score function weighted with visual sentiment score and textual sentiment score and an adaptive threshold approach can be used to enhance the validity of the given integrated sentiment scores.Finally,we adopt an exhaustive grid search technique to obtain the model parameters in the assumed sentiment score function in a brute force framework.The experimental results of the proposed sentiment recognition scheme on three benchmark datasets such as T-GIF dataset,GSO-2016 dataset and Adjusted-GIFGIF dataset shows that compared with the current short annotated videos sentiment classification method,the method we proposed has better classification effect,significantly improves the robustness of the videos sentiment classification,at the same time it can regress the emotional scores which describe the intensity of the emotions of the short videos,effectively learn and understand the sentiment expressed by the short annotated videos.