| With the rapid development of Internet technology,more accurate and efficient multi-modal retrieval systems are needed for handling massive multi-media data to support large-scale retrieval scenarios.In recent years,benefitting from the extremely low storage cost and high retrieval speed,hashing-based retrieval methods have attracted more and more attention.Compared with the uni-modal retrieval strategy,the multi-modal retrieval technique can capture the semantic correlations in the multi-modal data and integrate the multi-modal information to learn semantic-rich binary hash code.Although several existing works have made many remarkable achievements,there are still some problems to be solved: how to construct a multi-modal alignment space based on deep neural networks to effectively model multi-modal deep semantic correlations and reduce heterogeneous multi-modal gaps.Under the deep learning framework,how to preserve rich multi-modal semantic correlations into binary hash codes.Based on the existing feature-level multi-modal fusion strategy,how to further explore a more finegrained fusion process.It is difficult for coarse-grained label similarity learning strategies to model fine-grained semantic correlations between different items.To tackle these challenges,this thesis takes more consideration on deep correlation modeling to handle the multi-modal hashing retrieval task,which mainly includes the following two works:(1)In this thesis,an efficient Hierarchical Message Aggregation Hashing(HMAH)for crossmodal retrieval is proposed in the framework of teacher-student learning.In the teacher module,the hierarchical message aggregation network is introduced.Through the hierarchical aggregation process of semantic information across different modalities,a multi-modal complementary space is constructed to better align heterogeneous modalities and model the fine-grained multi-modal correlations.In the student module,this method supports efficient cross-modal retrieval by training a couple of lightweight hash functions for each modality.In addition,a cross-modal correlations knowledge distillation strategy is designed to transfer the modeled fine-grained multi-modal semantic correlations from the teacher module to the student module.The semantic representation capability of the hash functions in the student module can be enhanced by the guidance of finegrained correlation knowledge from the teacher module.In addition,the whole learning framework does not need to fine-tune the pre-trained backbone model,thus improving computational efficiency.The experimental results show that,compared with the most advanced deep crossmodal hashing retrieval baselines,the proposed method significantly improves retrieval accuracy and efficiency(On MIRFLICKR-25 K,NUS-WIDE and MS COCO datasets,retrieval accuracy is improved by 2.41%,3.7% and 4.74%,training efficiency is improved by 3.8～606.9 times,13.9～575.5 times and 3.2～244.6 times,test efficiency is improved by 15.5～19.9 times,13.0～28.0times and 12.6～18.2 times).(2)In this thesis,a Bit-aware Semantic Transformer Hashing(BSTH)is proposed to excavate the bit-wise semantic concepts to align heterogeneous multi-modal on the concept-level to perform multi-modal hash learning.Firstly,this method introduces Transformer architectures to learn bitwise implicit semantic concepts with a self-attention mechanism,and performs multi-modal semantic alignment based on the fine-grained concept-level,thus reducing the gaps among different modalities.Then,the concept-level multi-modal fusion is conducted to enhance the semantic representation capability of each implicit concept,and the fused concepts are further encoded into corresponding hash bits by bit-wise hash functions.In addition,to effectively supervise the bit-aware Transformer module,this method introduces a label prototype learning module to learn the prototype features of all categories that capture the explicit semantic correlations on the category-level by considering co-occurrence priors.Experimental results on three widely used multi-modal retrieval datasets demonstrate the superiority of the proposed method(On extended MIRFLICKR-25 K,NUS-WIDE and MS COCO datasets,retrieval accuracy is improved by 8.48%,5.08% and 5.43%). |
