| In the era of multimedia big data,the global data scale is exploding and the data types are also diversifying,among which text and image data are the most common.Text and image matching technology can help users find valuable information quickly and accurately from the huge amount of data,so it is very important to study data matching.In this paper,we study the problem of fine-grained correspondence in text and image matching tasks based on deep learning methods,and our main work and innovations are as follows:(1)To address the problem that the existing text matching methods do not integrate multigrain information such as words,phrases and sentences simultaneously,this paper proposes a text matching model based on multi granularity information interaction(MGII).The model firstly generates word representations with contextual information through a bidirectional longand short-term memory network,phrase representations with local information through a convolutional neural network,and sentence representations with global semantic information through a self-attentive mechanism and maximum pooling.Next,each word in the text is interacted with the information of the other text at three granularities by the interactive attention mechanism,and the results of the three interactions are aggregated to generate the final representations of the two texts.The experimental results show that MGII achieves better performance on the two text-matching datasets compared with models using a single granularity or combining two granularities.(2)Although the MGII model fuses the interaction information of multiple granularities of words,phrases and sentences to generate text representations and obtains better matching results,it neglects the syntactic structure information of the text.Specifically,the model first obtains the dependency relations between words through dependency syntax analysis,and designs two graph construction strategies to arrange the syntactic dependency relations into the semantic graph.Secondly,the semantic graph is encoded using graph attention network to generate a text representation with syntactic structure information.The experimental results show that considering the dependent syntactic relations can help the model capture the deep semantic relations of the text,thus further improving the accuracy of the text matching task.(3)Existing image-text matching methods have the problems of difficult correspondence between attribute words and visual objects,and difficult alignment between regional relations and relational words.In this paper,we propose a image-text matching model based on visual semantic cross-modal attention networks.The model first constructs a visual graph using the appearance features of visual objects and their spatial relationships,and a semantic graph using linguistic features and syntactic dependency relationships.Next,the visual and semantic graphs are encoded using graph convolutional networks to obtain node representations containing spatial and syntactic structural relationships,respectively.Then the fine-grained correspondence between regions and words is modeled by a cross-modal attention mechanism to generate the final representations of visual and semantic nodes.Experimental results show that the model proposed in this chapter improves 1.1%,0.7%,and 1.4% in metrics R@1,R@5,and R@10,respectively,compared with the best performing baseline model when retrieving images from text. |
