| Recently,multimodal data with related content or semantic has been spreading on the internet.For example,objects in images can be associated with words in natural language,while the entire image can correspond to a sentence.Mining such local/global semantic alignments is of great importance for processing and analyzing multimodal data,and is also a key to realizing multimodal perception and learning.Scholars have proposed a series of tasks to investigate the local and global semantic alignments between images and text,such as visual relationship detection,scene graph generation(local scene understanding)and image-text matching,visual question answering(global content understanding).To mine such semantic alignments between images and text,this thesis focuses on the research of scene graph generation(SGG)and image-text matching(ITM).Given an input image,SGG can generate a structured scene graph,composed of triplets <subject,predicate,object>,aligning the scene of an image with textual words.Given a pair of image and sentence,ITM can judge the relevance between them by analyzing whether the semantic of them are consistent.However,there are still some issues in SGG and ITM at present.On the one hand,previous SGG methods cannot fully exploit multimodal context.Intra-modal context can assist models to understand local content in different modalities,while inter-modal context can fulfill the semantic gap between different modalities.The lack of such multimodal context can lead to suboptimal multimodal representations,which may result in inaccurate local alignments between images and text.On the other hand,local noisy features can affect the global matching between different modalities.In ITM models,the matching score between an image and a sentence is inferred from local features.However,the existence of local noisy features can disturb the feature space,impacting the global semantic alignment between the image and sentence.Therefore,we carry out some researches to solve these problems.(1)To address the problem of lacking multimodal context,this thesis proposes a cross-modal graph attention network for scene graph generation.Building on knowledge graphs,this method aligns the visual nodes with the textual nodes in the knowledge graph to generate a scene graph,thereby achieving the goal of aligning the local semantics of text and the image.Specifically,We propose a cross-modal graph attention mechanism to facilitate the interactions between nodes in the scene graph and textual nodes in the knowledge graph.Meanwhile,graph transformer network is employed to broaden the respective field of knowledge nodes.Experimental results demonstrate that the proposed approach achieves superior performance compared to baseline models on the Visual Genome dataset,and can better achieve local semantic alignments between visual content and textual entities.(2)To address the problem of local noisy features and global misalignment between different modalities,this thesis proposes a hybrid focal attention network for image-text matching.By designing focal cross/self attention mechanism,our model can concentrate on useful multi-modal features during the process of modality interaction,thereby reducing the impact of noisy information.Specifically,we design a focal cross attention mechanism to alleviate the noisy crossmodal features by sharpening or smoothing the cross attention scores.Then,the designed focal self attention scores local features to distinguish their importance and reduce the attention to unimportant features.Experimental results demonstrate that the proposed approach achieves superior performance compared to baseline models on the Flicker30 k and MSCOCO datasets,and can better achieve global semantic alignments between entire images and textual sentences. |
