Research On Visual Sentiment Analysis In Web Images Based On Weakly Supervised Learning

Sentiment plays an important role in daily life,which helps people to express their thoughts.In addition,Visual Sentiment Analysis(VSA)system is one main goal of human-computer interaction.In recent years,many studies have promoted the rapid development of VSA methods and systems through large-scale emotion datasets.However,the annotation of large-scale data set is very expensive and time consuming for that sentiment in one image is subjective.While,there are a large number of images on the Internet,which can express emotion.It is possible to obtain the sentiment annotations of images with noisy label by the image search engine from the Internet,by using keywords(like happiness,sadness,etc.)as queries.However,it will lead to terrible performance,when directly using the noisy-labeled samples for training the VSA model.That is a major challenge that how to obtain the effective VSA model by the noisy-labeled samples.In addition,many studies have found that local information has significant contributions to VSA.Similarly,the annotation of affective region in the images is also extremely labor-intensive and costly.Meanwhile,the discovery of affective region is different from the traditional object detection for that not only the objects but also the background region can evoke the sentiment.So how to discover the affective region including the background is an important challenge.For the two problems mentioned above in VSA,we respectively proposed two methods: Weakly Supervised Network for VSA based on Attention Mechanism and Weakly Supervised Sentiment Region Discovery for VSA.Meanwhile,the task of discovering sentiment region can improve the performance of sentiment recognition.The main content and contributions of this article as follows:(1)Weakly Supervised Network for VSA based on Attention Mechanism: For the difficulty of labeling the emotion datasets,we proposed the framework NLWSNet: Weakly Supervised Network for Visual Sentiment Analysis in Noisy-labeled Web Images.In this method,we propose a novel attention which can suppress the negative impact of noisy-labeled images.That we can directly use the web images for training.Meanwhile,we introduce the Specific-Class Activation Map(SCAM)to learn the specific-sentiment local feature.Finally,we introduce the triple loss and center loss as the regularization for sentiment classification,which can minimize intra-class distances while maximize distances between samples from different categories.The experiment results show that the effect of the proposed method: the proposed method still works when the proportion of samples randomly selected from noisy training set reaches 25%(Flickr: 80.73%;Instagram: 77.05%).When the proportion of noisy training samples reaches 50%,we achieve the better performance than other attention mechanism about 20% in the recognition accuracy on dataset Flickr and Instagram.Meanwhile,the results of sentiment recognition on multiple public datasets Twitter??EmotionROI?Flickr?Instagram,are also better than the state-of-art methods about 2%-3%.(2)Weakly Supervised Sentiment Region Discovery for VSA: For the difficulty of labeling the affective region in emotion image samples,we proposed the method: Weakly Supervised Sentiment Region Discovery for VSA.Specifically,we propose the Region Proposal Network with Multiple Convolution kernels(MCRPN)module,which can generate a large number of affective candidate regions.Meanwhile,the candidate regions include both the object and background through multiple kernels.Secondly,we design an end-to end loss of Multiple Instance Learning(MIL)to discover the affective regions.The regions generated from the corresponding emotion are composed as positive bags and the regions from other emotion are composed as positive bags.The affective region is discovered through iterative learning.Finally,the feature obtained by affective region is viewed as local information and the feature obtained by traditional deep learning model is viewed as holistic information.The fusion of feature improves the accuracy of VSA.The experiment results show that the effect of the proposed method: on the task of sentiment recognition,we achieve the comparable results to the state-of-the-art method on public datasets Twitter??EmotionROI?Flickr?FI.On the task of discovering affective regions,we achieve the better result on the evaluation index F1-score and Recall than the state-of-the-art method.Meanwhile,the accuracy of sentiment recognition task is increased by 11%-13% by the assist of affective regions discovery task.(3)Design and Implement of the Prototype System of Weakly Supervised Learning for VSA in Web Images: We design and implement the prototype system of weakly supervised learning for VSA in web images.We design the interface of system by programming language Matlab.The Python programming language with both Tensorflow and Keras package are used to design and implement such system,which consists of four modules: data processing,emotion recognition,discovery of affective regions and visualization of result.The prototype system can verify the usability and effectiveness of the proposed methods.