| With the progress and development of social media and computer networks,multimodal data such as image,video,text and audio have become important information media for human perception of the world.Therefore,how to provide an efficient and flexible multimedia retrieval system to meet the needs of human beings for different modal data retrieval is of great significance.Cross modal retrieval algorithm is such an effective retrieval system,which users can obtain the corresponding retrieval results of other media types by submitting the query data of any media type.However,due to the inconsistency of distribution between data from different modalities,how to accurately measure the matching degree of different modal sample features is one of the main challenges faced by cross modal retrieval.Aiming at this challenging research field,this paper designs two cross modal retrieval algorithm models based on cross modal attention mechanism and graph convolution neural network respectively.Firstly,most of the existing cross modal retrieval methods rely on one-step reasoning to reveal the interaction between visual semantics,and thus lack the ability to locate the correlation between hierarchical fine-grained features by using multi-level information.To alleviate this issue,we propose a step wise hierarchical alignment network for image text matching(SHAN),which decomposes image text matching into a multi-step cross modal reasoning process.Specifically,we first implement local to local alignment at the fragment features level,and then perform global to local and global to global alignment at the context feature level.This progressive alignment strategy provides more complementary semantic clues for our model to understand the hierarchical relationship between image and text,so that the model can learn a more accurate cross modal matching measurement.Secondly,from the perspective of feature enhancement,we propose a heterogeneous memory enhanced graph reasoning network for cross modal retrieval(HMGR).On the one hand,we design a new two-path graph reasoning network structure,The global representation of the relationship enhancement is generated by graph reasoning on specific modalities for the local features extracted in each instance.In this way,we can fully mine the topological correlations between local fragments in visual and text instances,so as to have a deeper semantic understanding of the relationship between them.On the other hand,we focus on using semantic knowledge between instances to enhance the distinguishability of the finally learned feature representations,which is realized by introducing a joint heterogeneous memory network to iteratively store visual and text instance level information.By interacting with long-term context-level multimodal knowledge,we can learn a better semantic latent feature space to alleviate the heterogeneous differences between different modalities.The above two algorithms have carried out extensive verification experiments on the public image text retrieval datasets Flickr30k and MS-COCO,and video text retrieval dataset TGIF.Detailed quantitative and qualitative analysis of the experiment proved the advancement and effectiveness of the proposed methods. |
