| Recently,with the rapid development of Internet and the widespread popularity of social media,global image and video data have shown explosively growth,human society has entered the big data era of visual information.Driven by the artificial intelligence,how to make computers understand visual content like humans and realize automatic semantic analysis of visual data,have been hot research topics.Visual Captioning as a fundamental problem,has attracted wide attention from scholars at home and abroad.Visual Captioning is a task integrating two fields of computer vision and natural language processing,which aims to generate a coherent natural language description corresponding to the visual content of the given visual data.It has important practical values in many aspects such as image retrieval,human-robot interaction,and visually impaired assistance.At present,with the widespread application of deep learning technology in visual tasks,visual captioning has made some progress.However,due to the large semantic gap between vision and language modalities,the cross-modal semantic alignment between vision and language still needs to be studied.This thesis conducts an in-depth study on the “visual-linguistic” semantic alignment based on the reinforcement learning method,and explores the coarse-grained salient semantic alignment and the fine-grained structured semantic alignment methods respectively.The innovations are summarized as follows:To address the problem of coarse-grained salient semantic alignment,this thesis proposes a Word-Level Semantic Units Matching method,which designs a local semantic similarity evaluation mechanism based on the comparison of “visual-linguistic”semantic units,to dig the latent semantic correlation between salient visual objects and generated linguistic words,and fuses the local and global semantic similarities using the reinforcement learning theory to guide the model parameter update to achieve the object-level coarse-grained semantic alignment.Aiming at the problem of fine-grained structured semantic alignment,this thesis proposes a Topology-Level Multi-Graph Coupling method.It introduces the structured scene graph into the reinforcement learning framework to construct a multi-agent policy network and propose a novel graph-coupled reward function,which can explicitly measure the “visual-linguistic” graph correlation at both the node-level and topology-level,and mutually align them for each agent optimization to effectively realize the “visuallinguistic” structured semantic alignment in the form of graph matching.The thesis verifies the proposed methods on the widely used large-scale MSCOCO dataset.The comparison performances with existing methods and the ablation studies fully demonstrate the effectiveness and superiority of the proposed methods.The visualization results and analysis also prove the capacities. |
