| Accurate and comprehensive caption of image content is of great significance for tasks such as blind navigation and robot perception of the environment.There are serious consequences if a blind person or robot makes a wrong action due to an inaccurate image caption.Therefore,it is necessary to improve the accuracy and comprehensiveness of image caption through advanced means.However,image caption is a cross-modal task,which needs to overcome the difficulty of visual and text feature extraction and fusion,especially for application scenarios with low fault tolerance.In response to the above problems,this paper proposes a Transformer model(Multilayer View Transformer Model,MLV)based on multi-layer views for image caption.MLV utilizes the visual information of different granularities output by different coding layers,and first establishes the Layer by Layer Cross View Module(LLCV)to facilitate the construction of the overall language;secondly,in order to further utilize the different granularity of the encoder visual information,prompting the model to learn more semantic features,the establishment of a multi-layer average view module(Multi-layer Average View Module,MLAV).Finally,in order to prevent the loss of two-dimensional spatial information caused by the input of the pre-extracted features to the Transformer encoder,this paper extracts the relative position information in two-dimensional space.The experiment demonstrated that MLV achieved BLEU-1,BLEU-2,BLEU-3,BLEU-4,METEOR,ROUGE,and CIDEr-D scores of 82.41,67.27,52.68,40.45,29.68,59.52 and134.28,respectively,on the MS-COCO dataset,which were improved by 1.62%,2.2%,2.65%,2.93%,0.95%,1.22% and 0.74%,compared to the baseline method.The comparison from various aspects proved the superiority of MLV in image captioning tasks.Although MLV has considered the problem of cross-modal information fusion and achieved good results in image caption tasks,it cannot integrate visual information and text information well when captioning visual vocabulary and non-visual vocabulary and produce reasonable contributions for both types of lexical predictions.Therefore,this paper proposes an Adaptive Associated Attention Network Based on Vision-Text(A3-VT)for image caption.A3-VT first merges the attention and information matrix of visual information and text information respectively to obtain fusion attention and fusion text feature information,and then calculates the fusion output of the two.At the same time,in order to capture richer visual and text feature information,A3-VT uses a multi-head attention model.The experimental results demonstrated that the A3-VT model significantly improved the performance of image captioning by utilizing visual-text adaptive fusion attention and multi-head attention.On the MS-COCO dataset,the A3-VT model achieved BLEU-1,BLEU-2,BLEU-3,LEU-4,and CIDEr-D scores of 83.29,67.87,53.27,40.84,and 135.58,respectively,which were improved by 1.07%,0.89%,1.12%,0.96%,and 0.97% compared to the corresponding indicators of the method in Chapter 3.The other three indicators were also improved to varying degrees.The comparison with other methods verified the superiority of the A3-VT model. |
