Cross-modal Retrieval Method Based On Dependence Relationship Attention And Social Information

With the rapid development of Internet technology,multimedia data of different modalities have grown exponentially,and people have been unable to satisfy the original single-modal data retrieval methods such as image retrieval.Cross-modal multimedia retrieval has become an important research field of information retrieval..Cross-modal multimedia retrieval technology is to achieve successful matching between different modal data with similar semantic content.This paper mainly studies the problem of mutual retrieval of images and texts in the field of cross-modal multimedia retrieval.The task consists of two parts: known query images,retrieval of relevant texts;known query texts,and retrieval of related images.Based on the deep network model,this paper considers the different biases of image semantic representation in different regions of image text and the complex interactivity between social media data.It proposes two aspects to improve the performance of cross-modal retrieval.:(1)proposing a cross-modal retrieval method based on sentence-dependent attention enhancement,because different semantic phrases of sentences have different accuracy for expressing image semantics,by generating a sentence-dependent tree for sentences,in the original double-branch network model In the cross-modal retrieval model,attention mechanism based on dependency phrases is added to enhance the influence of important phrase combinations in the process of text representation learning.(2)Propose a cross-modal retrieval method for integrating social information.Because in social networking websites,due to sharing and communication between users,there is a complex social interaction between multimedia data,by establishing a social communication network between image data.The social information feature vector of the image is extracted,and based on the original two-branch network model,the social information network is integrated to construct a cross-modal retrieval method based on the three-branch network structure.The experimental results show that compared with other comparison algorithms,the retrieval performance of the proposed method is improved to some extent,and the effectiveness of the proposed algorithm is verified.