Information Retrieval Based On Multimodal Deep Hash Learning

Information retrieval is the biggest demand for people to access the Internet.In the past,the mainstream way was to input key text information in search engines to obtain relevant texts or images.However,with the appearance of the era of big data,multimodal information retrieval has become a non-negligible requirement,and it has become the focus and difficulty of research in the field of information retrieval.This thesis aims to design a model,which can realize mutual retrieval between any two modalities,and can simultaneously retrieve the data corresponding to all other modalities via the data of one modal,this thesis uses the flexible scalability of Gaussian-binary restricted Boltzmann machines(GRBM)in deep learning models,an adaptive modal deep hash model based on GRBM is proposed.And based on this research,combined with the idea of manifold learning,an improved model based on multi-graph regularization is proposed,the retrieval performance of the model is improved greatly.The specific research work of this thesis is shown as follows:1.Aiming at the problem that most existing deep hash models cannot be extended to more than two modal data,this thesis proposes an adaptive modal deep hash model(AMH)based on GRBM.AMH can adaptively adjust the structure according to the type of multimodal data and the number of modalities.The model is divided into two parts: deep feature learning and shared hash code learning.First,AMH uses the deep feature learning part to obtain the deep features of the multimodal data,then uses the deep feature as the input of the visible layers of the AGRBM.The hidden layer result obtained by training directly serves as the shared hash codes of multimodal data.AMH realizes mutual retrieval between any two modalities,and can simultaneously retrieve the data corresponding to all other modalities via the data of one modal.Experiments on the bi-modal dataset show that AMH can compete with the existing state-of-art cross-modal hash model in the MAP results;experiments on tri-modal and four-modal datasets prove the model can adaptively adjust structure according to multimodal data,obtain shared hash codes and realize multi-modal data mutual retrieval.2.Aiming at the problem that the GRBM ignores the manifold structure of date during the training process,based on the existing work and combined with the idea of manifold learning,an adaptive modal hash model based on multi-Graph regularization(AMH-G)is proposed.Based on AMH,plurality of neighbor graph matrices are constructed according to the deep features of the multimodal data,then an affinity matrix is constructed according to the labels of the data.Add all the matrices together,as the graph regularization matrix.In the hash code learning part,the training of the AGRBM is combined with the graph regularization matrix,so that the hidden layer can learn the distribution of the multimodal data while maintaining the geometric information of the data.Experiments on bi-modal dataset show that AMH-G surpasses the state-of-art method of baselines in MAP and PR curve results.Experiments on the tri-modal dataset show that the retrieval performance of AMH-G has a 15% improvement over the original model.3.A simulation system for multimodal picture retrieval was designed and implemented.The core of this system is based on the adaptive modal hash proposed in this thesis and its improved algorithm.The user can select a multimodal dataset,then import a picture of a certain mode to be retrieved into the system,and select the retrieval requirements,the system can retrieve other modal images according to the user's retrieval requirements.And the search results will be presented in the system interface.This system is a prototype system,if it can be applied to large-scale data,it can meet the user's various retrieval requirements and greatly improve the user's retrieval experience.