Research On Automatic Annotation, TAG Processing And Reranking In Image Retrieval

In recent decades, the number of online images has been growing explosively with the advances in digital photography, networking and storage technologies. Especially along with the emergence of Web 2.0 technology, a large number of user-contributed images have been shared and spread on social sharing websites such as Flickr and Instagram. Meanwhile, this proliferation of images on the Internet has highlighted the urgent need for effective image retrieval systems. Existing commercial image retrieval systems provide search service for users by relying on the textual information associated with the images. However, manual image annotation is a laborious and time-consuming process. Therefore, automatic image annotation has been extensively studied for image retrieval. On the other hand, for the images on social sharing sites, the user-provided tags are usually imprecise and incomplement to describe the real image contents, and they can also not be regared as reliable indexing keywords for image retrieval. Therefore, a fundamental study in image retrieval is to process these unqualified tags, especially to improve the descriptive power of the tags with respect to image contents. In addition, as only the textual information of images is exploited, the returned results by the exisiting image retrieval systems are usually unsatisfactory. Therefore, how to rerank the initial retrieval results based on the visual information of images has also become one of the most important issues in image retrieval.In this dissertation, we carry out a series of research endeavors for improving the performance of current image retrieval systems. The following key issues are thoroughly discussed:image annotation, tag relevance learning, tag recommendation and image reranking. The main research contents and innovations are listed as follows:(1) We propose a novel approach for image annotation, which simultaneously improves both phases of the neighbor-based methods. In the phase of neighbor search, different from existing work discovering the nearest neighbors with the predicted distance, we introduce a ranking-oriented neighbor search mechanism, which uses the learning to rank framework to directly optimize the relative ordering of labeled images rather than their absolute distance with respect to a given image. During the ranking process, we exploit the implicit preference information of labeled images and underline the accuracy of the top-ranked results. In the phase of keyword propagation, different from existing work using simple heuristic rules to select the propagated keywords, we present a learning-based keyword propagation strategy, which uses supervised learning techniques to learn a scoring function that can evaluate the relevance of candidate annotation keywords. The relevance is determined based on different kinds of relations between the candidate keywords and the neighbors of the new image. Extensive experiments on Corel 5K dataset and MIRFlickr dataset demonstrate the effectiveness of our approach.(2) We propose a ranking-oriented tag relevance learning method, which approaches the problem from a new perspective of learning to rank, and facilitates tag relevance learning to directly optimize the ranking performance of tag-based image search. Specifically, a supervised based procedure is introduced into the neighbor voting scheme, in which tag relevance is estimated by accumulating votes from visual neighbours. Through explicitly modeling the neighbor weights and tag correlations, our method effectively avoids the risk of making heuristic assumptions for the conventional unsupervised methods. Besides, our method does not suffer from the scalability problem for the conventional supervised methods. Extensive experiments on two benchmark datasets in comparison with the state-of-the-art methods demonstrate the promise of our approach.(3) We propose an image tag recommendation approach combining relevance and diversity. Previous methods on image tag recommendation are usually developed based on tag co-occurrence information. However, due to the neglect of the visual information associated with images and the semantic diversity among recommended tags, the recommendation results of previous methods often suffer from the problems of tag ambiguity and redundancy. To solve the above problems, our approach considers both the relevance and diversity of the recommended tags. Firstly, the approach builds the visual language model of each tag, and employs it to calculate the relevance between a tag and an image, as well as the visual distance between two tags. Then, according to the above calculations, a greedy search algorithm is proposed to find a tag set as the final recommendation, which reaches a reasonable trade-off between the relevance and diversity. Experiments on Flickr dataset show the proposed approach outperforms the state-of-the-art methods in terms of precision, topic coverage and F1 value.(4) We propose a hybrid relevant-diverse image reranking approach, which is a clustering-based reranking method and combines the strengths of two previous methods:the reciprocal election algorithm and the greedy search algorithm. Our approach selects several candidate representatives from the initial search results based on the reciprocal election algorithm, and employs a bounded greedy search algorithm to find the most relevant and novelty one as the cluster center. To compute the similarity between images, we fuse multiple features including color, shape and especially latent topic models, and thoroughly discuss the benefits of integrating different features. We evaluate our approach on a real-world Web image dataset and the experiment results suggest that our approach can improve the user satisfaction from different aspects of cluster recall, NDCG and F1 score.