Research Of Weakly-supervised Cross-modal Hashing Learning

With the continuous development of the Internet,people can easily access a large number of multimedia data.Due to the superiorities of low memory consumption and fast query speed,hashing learning has been widely used in similarity search of multimedia data.Especially,the application of multi-modal hashing learning in cross-modal multimedia data(such as images and texts)has attracted more and more attention.Cross-modal hashing learning has made a lot of achievements in recent years.However,the current cross-modal hashing methods still face many weakly-supervised problems,such as the insufficient label information,the incompatibility between feature learning and hash quantization processes,the inability to effectively identify new categories and so on,which limit the application of cross-modal hashing to more open scenarios.In this paper,more practical cross-modal hashing approaches are proposed to address different scenarios of weakly-supervised cross-modal hash learning and achieve efficient cross-modal data retrieval.The main works of this paper are as follows:(1)In order to achieve efficient cross modal hashing learning,this paper proposes a Ranking-based Deep Cross-modal Hashing algorithm(RDCMH).RDCMH firstly uses the feature information and label information of data to get semi-supervised semantic ranking lists.Secondly,to improve the semantic presentation of row features,RDCMH integrates semantic ranking information into deep cross modal hashing,and jointly optimizes the compatible learning parameters of deep feature representation and hashing function.Experiments on real multi-modal datasets show that RDCMH is better than other methods in cross modal retrieval applications,and it can well deal with semi-supervised hashing scenarios.(2)To further address the problems that the label information are insufficient andthe label spaces are also different,and existing hashing methods cannot effectively recognize the new categories which has not appeared in the training data.This paper designs a Cross-modal Zero-shot Hashing(CZHash)solution,which can effectively utilize both the unlabeled and labeled multi-modal data in different label spaces.CZHash first uses the label and feature information to quantify a composite similarity between samples.Then a unified objective function is proposed to achieve the compatible learning of deep features,category attribute space and hashing coding functions.CZHash further introduces an alternative optimization solution to jointly optimize these learning objectives.Experiments on multi-modal datasets show that our CZHash outperforms the related representative hashing methods both on efficiency and adaptability.(3)Both RDCMH and CZHash can only work on two modalities.However,they still face the problem of high computational complexity when the number of modalities is more than 3.Besides,they do not further consider the incompleteness and insufficiency of labels.To address the above problems,this paper proposes a Weakly-supervised Cross-modal hashing algorithm(WCHash).Specifically,WCHash first uses the data features of all modalities to optimize a potential central modality.Secondly,an efficient multi-label weak label method is utilized to complete and enrich the labels of training data,and the semantic similarity between samples is measured based on the enriched label.Then the semantic similarity is used to maximize the correlations between each modality and the central modality,so that we can learn the hash functions for cross modal retrieval.Experimental results on real datasets show that WCHash is more efficient than the state-of-art cross-modal hashing methods.In addition,WCHash can significantly reduce the computational complexity of cross modal hashing on three or more modalities.(4)Finally,we consider a more flexible cross-modal hashing scenario,in which the correspondences between samples in different modalities are unknown,and the numbers of samples in different modalities are not the same.To solve these problems,this paper proposes a Flexible Cross-modal hashing method(Flex CMH)to learn effective hash codes from weakly-paired(or even completely unpaired)multi-modal data.FlexCMH firstly introduces a clustering based matching strategy to explore the structure of each cluster,to find the potential cross modal correspondence betweenclusters(and the samples therein).It further jointly optimizes the potential correspondence,the cross-modal hashing functions derived from the correspondence,and a hashing quantitative loss in a unified objective function.An alternative optimization technique is also proposed to coordinate the correspondence and hash functions,and to reinforce the reciprocal effects of the two objectives.Experiments on public multi-modal datasets show that FlexCMH achieves significantly better results than state-of-the-art methods,and it indeed offers a high degree of flexibility for practical cross-modal hashing tasks.