Semantics Consistent Adversarial Cross-Modal Retrieval

Cross-modal retrieval refers to a retrieval method that returns a modal query word and returns other modal data related to its semantics.The biggest difficulty in cross-modal retrieval is the “heterogeneity” problem caused by the different representations and distributions between different modalities,which leads to the fact that the similarities between different modal samples cannot be directly measured.This thesis solves the problem of “heterogeneity” by mapping different modal samples into common subspaces and then measuring the similarity of different modal samples in the common subspace.Specifically,this thesis did the following:For samples with labels,a cross-modal retrieval method based on semantic consistent adversarial network is proposed.Specifically,under the framework of adversarial learning: 1)finding the class center of each semantic class with different semantic tags in different modalities;2)under the same modal,minimizing the Euclidean distance of the same semantic sample with the class center;3)Under different modals,minimizing the Euclidean distance with the same semantic class center;4)for class centers,minimizing the distance between the class center and other modal and samples with the same semantic class.Comprehensive experiment were carried out on the Wikipedia dataset and the NUSWIDE-10 k dataset.The experimental results show the effectiveness of the proposed method in cross-modal retrieval.For parts of samples with labels,a semi-supervised adversarial cross-modal retrieval method based on graph constraints is proposed.Specifically,under the framework of adversarial learning: 1)for unlabeled samples,constructing a graph based on Euclidean distances between samples,expecting common subspace representations of similar samples to be similar;2)learning from labeled samples using traditional adversarial cross-modal retrieval methods;3)the unlabeled sample and the labeled sample learn the common subspace representation together in the anti-learning framework.The experimental results on the Wikipedia dataset and the NUSWIDE-10 k dataset show that the proposed method achieves a comparable result to that of the supervised learning retrieval method.