Cross-modal Retrieval With Limited Data Resource

With the emergence of huge amounts of multimedia data,cross-modal retrieval becomes a hot research topic.So far,most cross-modal retrieval methods depend on large-scale and labeled training data,which is expensive.In real world,data resource is usually limited.Specifically,we focus on two sub-settings: on one hand,the amount of data is limited,on the other hand the labels of data is unavailable.In this paper we have conducted research on those two settings and made progress respectively.When data amount is limited,we propose a cross-modal transfer method.The contributions are two-fold.Firstly,we have to transfer modal-shared knowledge.We propose two-level transfer,which transfers both modal-specific and modal-shared knowledge by adversarial transfer.Secondly,the label space differs between source and target domain,which means employing homogeneous methods may result in negative transfer.We proposed progressive transfer based on curriculum learning.We use large-scale dataset as source domain and three small-scale dataset as target domain to conduct transfer experiments,and the results could demonstrate the effectiveness of our approach.As for the unsupervised setting,we propose a method from the perspective of learning with noisy label.Previous state-of-the-art method typically extract pseudo labels through pretrained networks and then fit them.However,pseudo labels are extremely noisy.Therefore,we propose a novel method to tackle the issue: two hashing models are trained simultaneously,screening those pairs with noisy labels and then teaching each other only using picked cleaner supervision in a co-training manner.Empirical results on real-world datasets have proved the effectiveness of our approach.