| Image caption is an comprehensive task that combines 'Computer Vision' and'Natural Language Processing'.It is a challenging task mainly due to the complex relationship of picture content and the diversity of natural language descriptions.Inspired by recent work in machine translation,An encoder-decoder based algorithm was proposed that improved the performance obviously.This algorithm uses a convolution neural network to extract visual feature(encoding)and generates captions using a sequential recurrent prediction model(decoding).In this paper,some shortcomings of the existing image caption methods are summarized,and the correlation constraints are added to the traditional attention-based image caption algorithm to explore the coherence and relevance of visual information and semantic information Simultaneously.The coherence is introduced to maximize the probability of generating the next word according to previous words and the current visual contents which can ensure the correctness of the grammatical logic of the generated caption.And the relevance tries to explore the relationship between the semantic and visual contents in a semantic-visual embedding space which can ensure that the content of the title description and picture content is more relevant.The method proposed in this paper is more general.The performance of our model is validated with two benchmark datasets: Flickr30 k and MS COCO.And we evaluate the performance of our algorithm using BLEU-1,2,3,4,Meteor,Rouge-L,and CIDEr scoring matrices.The experimental results show that our approach can improve the performance of image caption method without considering the relevance observably.The generated image captions are more flexible and diverse. |
