Deep Compact Coding For Multimedia Nearest Neighbor Search

Recently,with the rapid development and popularization of the Internet and computer technologies,large volume multimedia data with ever-increasing diversities are preva-lent in modern search engines and social networks,which introduce great challenges to multimedia data index and search.Due to the low storage and computation cost,compact coding has been widely used to manage and analyze multimedia data.In this thesis,we provide a comprehensive study on the compact coding for multimedia similarity search.Concretely,we provide novel methods for unimodal similarity search and cross-modal similarity search based on hashing and quantization techniques.We also study the security problems for existing deep learning-based compact coding methods.We can summarize our main research as follow:(1)For unimodal supervised similarity search,we design a two-stream deep hashing method,which can learn class-specific centers for different classes and can also deal with multi-label applications efficiently.By learning hash codes and class-specific centers simultaneously,our methods can greatly reduce the intra-class variations and generate promising hash codes.(2)For unimodal unsupervised similarity search,we propose a semantic-structure based unsupervised hashing method.Note that recent supervised hashing methods,which usually construct semantic similarity matrices to guide hash code learning using label information,have shown promising results.However,it is relatively difficult to capture and utilize the semantic relationships between points in unsupervised settings.In this thesis,we empirically study the deep feature statistics and propose a novel method to construct semantic structures under unsupervised settings.The learned se-mantic structures enable us to adopt the typical loss functions for supervised hashing and greatly improve the search accuracy.(3)For cross-modal supervised similarity search,we propose a novel deep cross-modal hashing method to generate compact hash codes through an end-to-end deep learning architecture,which integrates different types of pairwise constraints to encourage the similarities of the hash codes from an intra-modal view and an inter-modal view respec-tively.Moreover,additional decorrelation constraints are introduced to the proposed architecture.Experimental results show that our method enhances the discriminative ability of the learned hash codes.(4)For cross-modal supervised similarity search,we further propose a deep quantiza-tion approach,which is among the early attempts of leveraging deep neural networks into quantization based cross-modal similarity search.Our method explicitly formu-lates a shared subspace across different modalities and two private subspaces for in-dividual modalities,representations in the shared subspace and the private subspaces are learned simultaneously by embedding them to a reproducing kernel Hilbert space where the mean embedding of different modality distributions can be explicitly com-pared.Additionally,in the shared subspace,a quantizer is learned to produce the semantics preserving compact codes with the help of label alignment.By doing so,our method can preserve intra-and inter-modal similarities as much as possible and greatly reduce quantization error.(5)For the security of deep learning-based compact coding methods,we study the robustness of modern deep hashing models to adversarial perturbations.Recent stud-ies highlight the vulnerability of deep image classifiers to adversarial examples which also introduces profound security concerns for deep retrieval systems.In this the-sis,we propose a novel method of crafting adversarial examples for Hamming space search.We generate a variety of perturbations from different models,and show that the transferability of these perturbations greatly exists under different settings.Moreover,by combining heterogeneous perturbations,we further provide a simple yet effective method of constructing adversarial examples for black-box attacks.In conclusion,we provide four compact coding models,which can reduce the intra-class variation,bridge the multi-modal semantic gap,and reduce the quantization error.Moreover,we also propose an adversarial example generation method for exist-ing deep hashing algorithms.Theoretical analysis and extensive experimental results demonstrate the superiority of the proposed models.