Research And Application Of Cross-modal Retrieval

With the explosive growth of multimedia data in recent years,such as image,text,video,and audio are pouring into the internet.The society has entered the era of big data.A large number of multimodal data contains an abundant economic value and social value.How to accurately retrieve from different modalities has become a hot issue for researchers and engineers.However,due to the heterogeneity gap and semantic gap among different modalities,which makes it is difficult to conduct cross-modal retrieval in the multimodal data directly.Due to its little storage cost and high computing efficiency,more and more researches pay attention to hashing cross-modal retrieval.Most of the existing hashing cross-modal retrieval methods map different original multimodal data into a common low-dimensional subspace.Then the latent feature subspace is quantization into hash codes by the hash function.However,there are still many problems to be solved in the field of hashing crossmodal retrieval.First of all,the semantic label plays an important role in feature representation.However,many supervised methods fail to make full use of semantic label information.Moreover,it is very important to keep the consistency of the original sample feature projection transformation for learning high-quality feature space.However,many methods only use collective matrix factorization and low-rank constraints to learn the common feature space,which will lead to a lot of feature loss.Further,many methods separate the feature extraction process and the hash function learning process.However,it will lead to the problem that the learned hash function can't reflect the features of the original data well.Finally,many methods do not fully consider the similarity retention inter-and intra-modal,which will seriously lead to the loss of the original features and other problems.In our thesis,with the help of collective matrix decomposition,subspace learning,residual preservation,neighborhood graph,promotions and improvements are proposed to solve the above problems.The main contents and contributions are as follows:In the local preserving hashing cross-modal retrieval method,the feature consistency of the original data before and after the transformation is well maintained by introducing the similarity residual preserving combined with the local similarity structure of the sample into the feature transformation process.Besides,the method also restricts the feature space after the original data transformation,to better maintain the local structure characteristics of the original samples and improve the identification of hash codes.Finally,the method combines the learning process of hash function with the feature extraction process to improve the quality of hash function learning.Aiming at the problem of feature loss in the process of original data transformation and learning high-quality hash function,a similarity preserving for cross-modal hashing retrieval method is proposed based on local preserving hashing for cross-modal retrieval.While minimizing the residual value before and after the feature transformation and combining the local feature structure of the original data,the method keeps the similarity between the original data modes by adding the triple graph constraint structure.For label information,we use linear regression to learn uniform hash code and apply label information to the hash function and local feature learning to improve the similarity retention ability of hash code and enhance the adaptability of the hash function.In the experiments,the two supervised cross-modal retrieval methods proposed in this paper are applied to three benchmark datasets and compared and analyzed with the state-of-art cross retrieval methods.The results show that compared with the existing cross-modal hashing retrieval method,the local preserving hashing for cross-modal retrieval and similarity preserving hashing for cross-modal retrieval proposed in this thesis can improve the retrieval accuracy and show better retrieval performance.