Research On Cross-modia Retrieval Based On Hidden Layer Semantic Correlation

With the development of Internet and multimedia technology, people are not satisfied to describe a document with single-modal data, documents contained of multi-modal content more and more frequently appear in the Internet, therefore more and more people pay attention to cross-media retrieval.Since direct mapping the different modal features into homogenous space will lose a lot of information, and the significance of the different modal features with same dimension is also different. This paper has proposed a new method to retrieve cross-modal data. The method is based on hidden layer semantic correlation. First, the idea of the deep learning was used for reference, through the deep network we can map the low-level features into the hidden layer features. The deep network is built with stacked auto encoder and error back propagation technology. And then correlating the hidden layer features of different modalities with canonical correlation analysis, so correlated hidden layer features of different modalities have the same dimensions, and have similar meanings. Next, the Softmax Regression is used to learn the semantic information of the correlated hidden layer feature of different modalities. Finally, calculating the similarity of semantics and features to carry on cross-media retrieval.In order to verify the effectiveness and feasibility of the method, experiment were carried on the Wiki data sets. Experimental results show that the hidden layer features acquired through the deep network have better results in the semantic classification and cross-media retrieval compared with the low-level features. The experiment also verified cross-media retrieval based on hidden layer semantic correlation has better results compared with cross-modal retrieval only based on semantics or features.