| Visual Captioning(VC)task is a fundamental and essential research area of cross Media Intelligent(CMI).It aims to automatically generate natural language text for images or videos to describe their contents.Recently,with the improvement of deep learning technology,Computer Vision(CV)and Natural Language Processing(NLP)have gained remarkable progress,and the VC as a research field involving both CV and NLP also made significant progress.Specifically,with the support of large-scale VC datasets,researchers design Visual Captioning Models(VCMs)based on deep learning encoder-decoder architecture.As a result,the proposed VCMs continuously push up the performance of the VC benchmarks.However,despite the impressive progress of VC,VC still faces shortcomings in three areas: model evaluation,model architecture,and supervision methods.Firstly,the standard VC benchmarks based model evaluation method cannot effectively evaluate the generalization ability of VCMs,which leads to ignoring the generalization ability of VCMs,and further hinder the design of VCMs.Secondly,existing VCMs tend to ignore the relation between visual feature channels in the visual encoding stage,which hurts the visual encoded features and thus reduces the quality of the captioning text generated by VCMs.Thirdly,establishing relationships between visual contents and nouns based on explicit supervision enhances VCMs.However,existing attention supervision methods ignore important local features of visual contents,which harms the finegrained and accuracy of captions generated by VCMs.To address the above issues,this thesis conducts studies for VC tasks,including exploring the critical factors for designing robust VCMs,improving the capability of visual encoding for VCMs and enhancing VCMs for describing visual objects.The innovations of this thesis are demonstrated as follows:(1)Proposing a maximum discrepancy competition based evaluation method for VCMs.This thesis experiments with Image Captioning Models(ICMs).In recent years,image captioning has gained excellent development with the support of large-scale benchmarks and deep learning techniques.However,the performance of ICMs has reached a bottleneck,and the recently proposed ICMs have only slightly improved performance with minimal differences.After observing this phenomenon,an important question emerges: " what about the performances of the recent ICMs achieved on in-the-wild images?" To clarify this question,this thesis compares existing ICMs by evaluating their generalization ability.Specifically,we propose a novel method based on maximum discrepancy competition to diagnose existing ICMs.Firstly,establishing a new test set containing only informative images selected by adopting maximum discrepancy competition on the existing ICMs,from an arbitrary large-scale raw image set.Secondly,a small-scale,low-cost subjective annotation experiment is conducted on the new test set.Thirdly,the generalization ability of the participating models is evaluated based on their performance in the new test set.Finally,the experimental results of the generalization ability evaluation are analyzed to obtain the key factors affecting the ICMs and summarize the potential research directions of ICMs.(2)Proposing a two-stream self-attention mechanism based deep learning model for image captioning.Recently,self-attention mechanism based encoder-decoder models dominate the image captioning area.However,most of them only focus on establishing relationships among spatial tokens when performing visual encoding,while the relationships among visual feature channels are neglected.Since different feature channels in visual features are believed to represent different visual objects,the neglect of channel-wise encoding hurts the performance of the nouns and adjectives about visual objects for ICMs.To alleviate this problem,a dualstream self-attention network is proposed in this thesis.Specifically,we design a dual-stream self-attention module with parallel blocks that encode visual information based on spatial and channel tokens.Meanwhile,to efficiently get effective channel-wise visual encoding features,we propose a group self-attention block with linear computational complexity as the channelwise visual coding block of the dual-stream self-attention module.The dual-stream selfattention module effectively improves the performance of the proposed ICMs by enhancing its visual encoding capability.(3)Proposing an informative attention supervision method for video description.Existing attention supervision methods explicitly guide VCMs to focus on relevant content of visual input when generating captions,thereby improving the accuracy and interpretability of VCMs.However,existing attention supervision methods often ignore small but informative visual regions because they are not considered to require attention according to the Intersection-overUnion based attention region sampling method used in existing attention supervision methods.On the other hand,the standard attention loss function of existing attention supervision methods forces VCMs to focus equally on all relevant visual regions when generating captions.Therefore,it makes it difficult for VCMs to concentrate on informative regions more relevant to the words and reduces the quality of the captions generated by VCMs.Furthermore,it causes VCMs difficulty focusing on informative regions that are more important to the captions,thereby reducing the quality of the captions.To alleviate the above problems,this thesis proposes an informative attention supervision method containing a proposal based attention groundtruth sampling method and a group based weak grounding supervision.The proposed proposal based attention groundtruth sampling method treats small regions that overlap with grounding labels as positive regions.In contrast,the group based weak grounding supervision allows the VCMs to assign attention to the positive regions dynamically.The proposed attention supervision method can be directly adopted to existing VCMs and improve the quality of captions generated VCMs without increasing the inference cost.In summary,this thesis first proposes a new model evaluation method,which complements the existing benchmarks based evaluation method.It reveals the critical factors of the generalization ability of ICMs,including visual coding,model architecture and attention mechanism.Based on the model evaluation results,this thesis also provides potential directions for improving the performance of VCMs.Therefore,this thesis proposes a visual encoding module that efficiently and effectively obtains better visual coding features and uses the visual coding features to improve the performance of ICMs.And,this thesis finds small but informative regions essential for enhancing the VCMs in terms of accuracy and fine-grained granularity from attention supervision. |
