Multi-Task Joint Optimization For Visual Sentiment Prediction

With the rapid development of multimedia technology,visual contents like images and videos have become a kind of mainstream media in the social network,increasing users tend to express their opinion or sentiment via images and videos.How can the machine understand human sentiments has attracted more and more attention.As an important topic of the field of artificial intelligence and computer vision,affective computing is of great significance for analyzing the image content accurately.Bridging the affective gap between the low-level information and subjective semantics is a key to recognize the affective content and establish the sentiment mapping that is consistent with human cognition.Numerous existing methods have been developed for identifying one dominant sentiment from the global view of images.However,there are two main limitations: first,humans' emotional responses to images can be determined by local regions,while most existing methods employ convolutional neural network to learn feature representations only from entire images;second,the sentiment is naturally complex and ambiguous due to the coexistence of different sentiment in the same stimuli,while the previous methods treat the problem as a single-label learning problem.In this paper,we propose to address the above problem via multi-task joint optimization framework.For the first problem,we propose a weakly supervised coupled network for weakly-supervised sentiment detection and classification tasks.We first detect the regions that evoke sentiments,which are then combined to learns representation for visual sentiment analysis.The detect and classification branches are unified in an end-to-end framework,which only requires image-level labels,thereby significantly reducing the annotation burden.Extensive experiments on several affective datasets demonstrate that the proposed method performs favorably against the state-of-the-art methods for visual sentiment analysis;In addition,we address the ambiguity problem via label distribution learning and develop a multi-task deep framework by jointly optimizing classification and distribution prediction.Since the majority voting scheme is widely adopted as the ground truth in this area,and few datasets have provided multiple sentiment labels.We further improve the generalization ability by exploiting two weak forms of prior knowledge to generate emotional distribution for each category.The experiments conducted on the public affective datasets demonstrate the proposed method outperforms the state-of-the-art approaches.We also show the resemblance to human perception of visual content with the support of a large-scale user study.