Multimodal-based Supervised Learning For Image Search Reranking

Along with the growth of image sharing Web sites, such as Flickr and the proposal of the concept of "Internet+", a lot of images are sprung on the network. Image retrieval gains its popularity in our daily lives. How to retrieve the images in which people are interested from those is becoming an issue which needs solving urgently. However, the performance of general search engines is less than satisfactory. To tackle this problem, image search reranking is proposed to enhance the performance of search engine.The aim of image search reranking is to rerank the images obtained by a conventional text-based image search engine to improve the search precision, diversity and so on. Current image reranking methods are often based on a single modality. However, it is hard to find a general modality which can work well for all kinds of queries. This thesis proposes a multimodal-based supervised learning for image search reranking. This method employs different modalities to rerank the images which return by the search engine.This dissertation revolves around how to efficient use of multimodal to improve the performance of image search reranking and mainly discusses the selection of image visual features, the calculation of image similarity, the calculation of generating features of images and the weight of generating features of images. The main contents are listed as follows:(1) For each image which is in the initial list, we utilize the following visual features:HSV Color Histogram, RGB Color Histogram, Block-wise Color Moment, Color Correlogram, Edge Distribution Histogram and Wavelet Texture.(2) For different modalities, different similarity graphs are constructed and different approaches are utilized to calculate the similarity between images on the graph.(3) Exploiting the similarity graphs and the initial list, we integrate the multiple modalities into query-independent reranking features, namely PageRank Pseudo Relevance Feedback, Density Feature, Initial Ranking Score Feature, and then fuse them into a 19-dimensional feature vector for each image.(4) The supervised method is employed to learn the weight of each reranking feature.(5) In our experiment, we evaluate our method on MSRA-MM dataset. We employ widely used Normalized Discounted Cumulative Gain (NDCG) to evaluate the performance of our method. To demonstrate the effectiveness of the proposed multimodal based supervised learning method, we compare it with three state-of-the-art approaches. The baseline includes Pseudo Relevance Feedback Reranking, Bayesian Visual Reranking. We employ different initial lists to analysis the effect of different initial lists. The experiments constructed on the MSRA-MM Dataset demonstrate the improvement in robust and effectiveness of the proposed method.