Ulti-modality Fusion In Internet Image Search

Encountered with big data, multi-modality characteristics of internet image search, image-search-image, text-search-image and text-image-search-image systems are all unsatisfying. In order to retrieve as many and as comprehensive images as possible, it’s necessary to study on multi-modality fusion in internet image search. Internet mage search is a kind of multi-modality learning problem in essence. Many algorithms and ideas on them have emerged---vector quantization or co-occurrence model, machine translation model, relevance model, structure model with class information, multi-label learning, complementary multi-modality fusion, multi-modality fusion based on matrix factorization, multi-modality fusion based on harmoniums, multi-modality fusion based on alignment learning, multi-modality joint learning, multi-modality learning on agreement, and multi-modality learning driven by big data. Based on their pros and cons, requirements for the design of our multi-modality learning models were put forward.A model which diffuses and superposes both documents’ and terms’similarity matrices through document-term matrix to learn the documents’semantic similarity matrix had been applied to mutual reinforcements of multi-modality similarity matrices. The difference of its application to multi-modality learning was analyzed, and the multi-modality fusion model supplemented by intra-modality high-order similarity reinforcement was put forward. The multi-domain similarity fusion algorithms which not only consider mutual reinforcements of multi domains but also consider the reinforcement on correlations between domains were analyzed, and combined with the additive characteristic of multi-modality similarity fusion, an idea which uses the alignment between modalities to reinforce correlations between modalities was proposed. And to get the matched correlations between modalities, a statistical model with the alignment between modalities as the optimization objective was put forward with analysis of its analogy to the canonical correlation analysis. A lot of experiments were done to study their behaviors and validities in the multi-modality image search application.Kernel matrix is capable to describe manifold, and can map data from multi modalities to the similarity space for comparison. Represented by the kernel matrix, the Markov field’s diffusion and its manifold characteristic, and the alignment between two fields, can all be described by the circuit network with introduced electric potential. Single-modality search can be represented by the circuit network with sources, which is equal to spectral clustering. The circuit network has its explanation in Hilbert space. Both PageRank and manifold ranking can be represented by circuit networks, and the fast iteration algorithm of the circuit network with sources can be deduced reversely. Multi-graph fusion model based on the circuit network was built, which can be represented by the regularized optimization and can be further extended. Experiments verified the effectiveness and advantage of using the circuit network to realize the multi-modality fusion.The circuit network model was theoretically explained by the Poisson equation, and rationalities of both the fast iteration algorithm of the circuit network model and the intra-modality high-order similarity reinforcement were explained by the inhomogeneous heat-conduction equation. Based on the analysis of multi-scale diffusion, multi-scale spaces on manifold were analyzed. By referring to the solution to fixing the fault of signal truncation in traditional signal processing, the multi-resolution row-nearest-neighbor filtering methods of the similarity matrix were put forward. The advantage of the partial differential equation in adding boundary conditions conveniently was employed to add the reinforcement of alignment between modalities into the multi-modality fusion model based on the circuit network.The technology roadmaps and the four contributions were summarized, and the future work was prospected.