| Image has become one of the most popular ways for people to express their sentiments on Internet social platforms,as more and more social media users use them as carriers to express views and share experiences.Image sentiment analysis is one of the vertical subdivisions of computer vision,which has important research value and application prospects.For example,it can predict the trend of public opinion,measure economic indicators,and monitor sentiment states in real-time.However,there are still some problems in the existing work of image sentiment analysis,such as the complexity of image content and the scarcity of high-quality sentiment images.In addition,the sentiment content of existing image sentiment analysis methods is not targeted enough,and the sentiment content contained in the image is not deeply explored.Therefore,this paper proposes an image sentiment analysis based on self-supervised sentiment region localization and self-attention mechanism.The self-supervised sentiment region localization network was designed to accurately locate the local area containing strong sentiment in the image,to alleviate the difficult problem of model training.The self-supervised sentiment region localization network was optimized to improve the quality of image sentiment semantic representation and accurately depict image sentiment information.A double self-attention module is introduced to model image context association and complete image sentiment analysis.The main works of this paper are as follows:(1)Image sentiment analysis model based on self-supervised sentiment region localization and ensemble learning: Aiming at the problem that details in the image are not mined effectively,a sentiment region localization network is constructed based on self-supervised learning to locate the local area containing strong sentiment in the image and analyze the area with a strong sentiment.Finally,the results of five deep learning networks are ensemble to obtain the image sentiment analysis results.The proposed model is superior to most mainstream baselines and does not require a large number of samples for training.Secondly,the self-supervised sentiment region localization network is constructed to accurately locate the local area containing effective sentiment,to better complete the sentiment analysis of the image.Through ensemble learning,the advantages of different networks are brought into play to realize the deep fusion of image sentiment semantics.(2)Image sentiment analysis model based on contrastive learning and knowledge distillation sentiment region localization: Based on the work(1),an efficient heterogeneous network is selected to construct a self-supervised affective region localization network.The ideas of knowledge distillation and contrastive learning are introduced,and the results of sentiment analysis are used to guide the sentiment region localization module,dig into the sentiment semantics in the image deeply,and improve the performance of the localization network.The results of image sentiment analysis are used to guide the learning of the positioning network,to make the positioning more accurate,and the more accurate positioning will promote the performance of image sentiment analysis better,and the two encourage each other to promote the performance of the model.Finally,a heterogeneous feature fusion module is designed to integrate the sentiment semantics from the heterogeneous self-supervised sentiment region localization network to complete the image sentiment analysis.The experimental results of the proposed model are better than most mainstream baselines and perform well in the Twitter I dataset.The model optimizes the localization performance of the network by knowledge distillation and uses the contrastive learning feature to have stronger sentiment discrimination.At the same time,it realizes the fusion of heterogeneous sentiment features and effectively improves the quality of image sentiment semantic representation.(3)Self-attention-guided distilling self-supervised sentiment region image sentiment analysis model: Based on the work(2),a double self-attention mechanism is introduced to obtain attention-weighted feature maps to obtain the internal correlation between local regions,to extract more adequate sentiment semantics.In addition,the heterogeneous feature fusion module is replaced by the self-supervised regional positioning network,and the secondary sentiment region positioning is carried out to further improve the accuracy of sentiment region positioning,and to improve the accuracy of image sentiment analysis.The prediction accuracy of the proposed model in the Twitter Ⅰ and FI datasets is 94.7% and 81.7% respectively,which is better than all mainstream baselines.The model adaptively integrates the global and local semantic features of images by using the self-attention mechanism to help the model understand the sentimental semantics in images better.At the same time,the secondary self-supervised sentiment region localization makes the model more accurate in sentiment region localization.Main innovations:(1)The self-supervised sentiment region localization network was proposed to enable the model to learn the sentiment semantics of the image by itself,locate the area containing strong sentiment in the image,and effectively improve the sentiment analysis ability of the model;(2)Contrastive learning is used to improve the image sentiment representation,knowledge distillation is introduced to improve the network sentiment region localization performance,and the image sentiment content was described better;(3)Based on the optimized self-supervised sentiment region localization network,double self-attention modules are added to establish image context association and complete high-quality image sentiment analysis. |
