Research On Text And Image Cross-media Retrieval Based On Decision Tree Hash

With the development of computer technology,mobile network and intelligent devices,data can be collected,processed and stored in any scene.These data are various in types with diversity form of structure.How to accurately and quickly retrieve the multimedia information corresponding to the topic from such data is a challenge to us.There are three defects in the current retrieval system:1.Today's search engine retrieval modal is based on keywords and web page label matching to retrieve,that is,each information source of needs to artificial tagging labels,which is not sustainable for explosive mass data;2.The retrieved contents is singleness based on the keyword matching,which could not realize the modal of cross-modality,such as image to retrieval text,text to retrieval image and so on;3.Intelligence cannot meet the requirements of the human,in other words,the search engine can only distinguish the short keywords.For large paragraphs and sentences,the search engine cannot understand the meaning of the semantics,and the information retrieved from the search engine is often different from the people's meaning.To solve the above problem,this paper proposed a cross-media retrieval method based on text and image,in the processing of feature extraction,the media of video could transform into images via scene segmentation and frame processing.Audios can be transformed into text to a certain extent via speech recognition and scene analysis techniques.Therefore,the research for cross-media retrieval based on text and image is of great significance.The paper first introduces the development of cross-media retrieval in recent years,including feature extraction,single-media retrieval scheme,cross-media retrieval scheme and model.Then introduces the model and main innovation of our work,proposed a new cross-media retrieval method named cross-media retrieval based on decision tree hash,which includes two kinds of feature extraction based on neural network,using CCA algorithm to reduction dimension and dimensionality unification,then using sub-modular algorithm for the hashing binary code inference problem.Finally,we learn hash functions by training boosted decision trees to fit the binary codes.In this way,two different modalities are mapped to the unified feature space,and via quantizing by hash functions transform it to a hash space,it ensures the measurability between different media,and realizes the mutual retrieval of images and texts.The proposed CMDTH cross-media retrieval framework in this paper solves the following problems:1.It realizes the cross-media retrieval of images and text in function;2.Using CCA algorithm to joint reduce the dimensions could retained the correlation of images and texts,and can effectively solve the "dimension disaster" of the underlying features of heterogeneous data.3.The two step quantization strategy based on decision tree hash can preserve the topic information of the features in the common space by the generated hash code;4.Hash inner product,which is like Hamming distance,using for measuring the correlation between data can effectively reduces the time complexity in retrieval process.In this paper,Wiki,COCO and NUS-wide are used as experimental data sets.And adopt deep learning open source framework Caffe and natural language processing tool NLTK for feature extraction.The image using AlexNet and text is skip-gram.The following experiment is doing on MATLAB,including CCA algorithm for dimensionality reduction,the way to learn hash codes and the evaluation indexes.We compare three evaluation indexes with other methods,and the results shows that our method has better performance in these three indexes.