Research On Cross-modal Retrieval Based On Matrix Factorization

Due to the rapid development of PC Internet technology,social network,mobile Internet and other information technologies,a large number of multi-modal data,such as image,text,audio,video and so on,have emerged in the cyberspace of information exchange and media communication.How to effectively conduct semantic correlation analysis and similarity measurement for these multi-modal data and realize the mutual matching between modalities has become a new research hotspot in the field of artificial intelligence.Although multimodal data has similar semantics,it usually has different physical meanings,different dimensions and different probability distributions,which brings great challenges to the matching between modalities.Low-rank approximation of matrices is one of the most common methods in machine learning.Non-negative Matrix Factorisation(NMF)has attracted more and more attention because the non-negative constraints of the extracted factors can enhance the physical interpretation.In addition to the fact that NMF can significantly reduce the data dimension and improve the robustness of the solution,it can also utilize the learned dictionary to fully represent the local and internal structure of the data.With the strong learning ability of NMF,many improved versions of NMF have been successfully applied to cross-modal retrieval.Moreover,hash technology has been successfully applied in cross modal retrieval due to its low storage requirements and impressive retrieval efficiency.Toward this end,this work conducts an in-depth study on modal similarity search in virtue of matrix factorization technology.The main research results are as follows:1.We design an unbalanced collaborative semantic convex matrix factorization model(SCMF).The main idea is to extract the intermediate-level features for the higher dimension modal data by a classical non-negative matrix factorization model,rather than directly extract the higher-level semantic features,so as to prevent the loss of useful feature information.For the lower dimension modal data,the high-level semantic features can be extracted directly by semantic convex matrix factorization model designed by combining semantic information.Then,the extracted middle-level features of the high-dimensional modality are mapped to the common semantic space,so as to achieve the similarity measurement between different modalities and achieve a better cross-modal matching performance.2.We propose a multi-modal graph regularized smooth matrix factorization hashing framework(MSFH).A parameter-controlled smooth matrix is inserted into the collaborative matrix factorization model to realize the sparse of both the dictionary of each modality and the extracted common features.In addition,the similarity graph between modalities is reconstructed by symmetric non-negative matrix factorization to improve the accuracy of modal matching in the case of unsupervised learning.Then the extracted common features are binarized by learned hash functions,so that the similarity search between modalities can be realized quickly.3.A coadjutant discrete matrix factorization hashing model(DMFH)is proposed for cross-modal matching.Based on the advantage that the geometric structure information of each modality contains stronger semantic discrimination information,the model performs matrix factorization on the nearest neighbor similarity graph of each modality,directly extracts the discrete common hash feature representation for all modalities,and overcomes the quantization loss caused by hash relaxation.With Stiefel manifold,a simple and novel updating algorithm is designed to directly learn the discrete hash codebook in a closed form,which not only reduces the computational complexity,but also improves the accuracy of similarity search between modalities.4.Aiming at the defect of lack of semantic label in DMFH model learning,DMFH is extended into a supervised discrete matrix factorization hashing model(SDMFH),which firstly constructs the similarity matrix between modalities by semantic tags,and factorizes the similarity matrix to extract the common discrete hash representation of multiple modalities,and then the semantic label is regressed to the extracted discrete hashbook,so as to strengthen the discriminating ability of the learned discrete hashbook.Experimental results on existing international public datasets demonstrate the effectiveness of the proposed algorithm.