Research Of Cross-media Retrieval Based On Neighborhood Reversibility

To better express rich information content, people used to group a variety of multimedia together to express the semantic information. Therefore, the research of cross-media retrieval is of great significance for understanding of the digital media and multimedia retrieval.This paper makes a deep study of neighborhood reversibility problem in cross-media retrieval, and analyzes the sensitivity of cross-media retrieval to the number of neighbors and neighborhood relationship in single-modality multimedia retrieval. Then we design a cross-media retrieval framework, named LE-KNN framework. Based on this framework, we introduce two methods to enhance the neighborhood reversibility between different modalities. Also, we propose a novel method to adaptively select the number of neighbors for each modalities. Finally, neighborhood reversibility is introduced to build the commonly shared semantic representation. The main contributions in this paper are listed as follows:(1) Based on semantic association, we build a semantic correlation model and construct a cross-media retrieval framework, named LE-KNN. Then two methods, CDM algorithm and neighborhood reversibility verifying algorithm, are introduced to enhance the neighborhood reversibility between two multimedia documents. By improving the neighborhood relationship between multimedia documents, the accuracy of cross-media retrieval can be improved.(2) We study the sensitivity of the number of neighbors to cross-media retrieval, and propose a method to adaptively select the number of neighbors. Through the distances between multimedia objects, we figure out an appropriate number of "real" neighbors for each multimedia document to solve the problem of sensitivity between number of neighbors and accuracy in cross-media retrieval.(3) We find that different modalities of multimedia documents have different accuracy in corresponding retrieval framework, while in actual cross-media retrieval task, we never have taken account of this problem. Aiming at taking full use of this information, we apply the neighborhood reversibility to figure out the "real" neighbors, then calculate the accuracy for each modality of multimedia. According to the accuracy of different modalities, feature vectors in different modality will get different weight when calculating the neighborhood relationship matrix in the construction of multimedia documents’ feature vector. Therefore, the accuracy of cross-media retrieval would be improved.