Sentimental Style Image Captioning Based On Reinforcement Learning And Attention Mechanism

Image Captioning which is the intersection of natural language processing and computer vision,aims to automatically generate corresponding text Captioning for a given image.Compared with objective image captioning that lack sentiment or language style,Sentimental style image captioning are more in line with human needs for image captioning.However,Sentimental style image captioning faces problems such as low content captioning accuracy and lacking diverse sentimental style.To this end,this paper explores how to integrate Sentimental style into image captioning on the basis of accurately describing image content,so as to realize Sentimental style image captioning.The main research contents include:(1)In order to make full use of the multi-level information in the image and improve the accuracy and generalization of the image captioning model for content captioning,an image captioning generation model based on reinforcement learning and multi-level attention mechanism is established.The model introduces a multi-level attention module(XI-P),which focuses on different levels of features according to the state of the generated sentence,and achieves the purpose of effectively utilizing multi-level information by fusing the multi-level attention features.In addition,in order to solve the problem of inconsistency between the optimization index and evaluation index of the image captioning model,a reinforcement learning fine-tuning module is introduced,which guides the model to generate captioning sentences through the policy network and value network with stronger decision-making ability.The experimental results on the MSCOCO dataset show that after introducing the multi-level attention module and the reinforcement learning fine-tuning module,the results of the three evaluation indicators of BLEU-1,BLEU-4 and CIDEr can be improved by 3.1,3.0 and 11.9.(2)In order to solve the problem of the lack of diverse Sentimental styles in the image captioning text,and the mismatch between the existing reinforcement learning optimization strategy and the Sentimentalstyle image captioning evaluation index,an Sentimentalstyle image captioning based on reinforcement learning and attention mechanism is established Model.The model encoder introduces a knowledge association module and a style memory module.The knowledge association module solves the problem that the Sentimental style of the generated text does not match the objective content by memorizing the similarity between the generated captioning and the real captioning.The style memory module solves the problem of lack of diverse Sentimental styles in image captioning by learning the characteristics of Sentimental style expressions in the multi-type Sentimental style corpus.The model decoding module embeds a multi-level attention module(XI-P)to improve the text decoding ability of the model.On this basis,the model solves the problem of inconsistency between existing reinforcement learning optimization strategies and Sentimentalstyle image captioning evaluation indicators by improving the optimization strategy in the reinforcement learning fine-tuning module.The experimental results on the Flickr Style10 K and Senti Cap datasets show that after introducing the knowledge association module,style memory module and reinforcement learning fine-tuning module,the evaluation of the four types of Sentimental style image captioning models on the three indicators of BLEU-1,BLEU-3,and CIDEr As a result,the average can be improved by about 3.25,1.6 and 5.4,the evaluation results on the PPL index are reduced by about8.3 on average,and the accuracy of Sentimentalstyle is about 95.75% on average.(3)According to the Sentimental style image captioning model proposed in this paper,a demonstration system that can automatically generate sentimental style image captioning for images is realized.The system mainly includes user login module,image captioning generation module,Sentimental style image captioning generation module and other related modules.Through this system,users can generate objective captioning by uploading images and generate Sentimental style image captioning according to the selected Sentimental style.