Research On Image Search Enhancing Using Object Semantics

In the age of Internet, search engines are changing the style of our daily life and this ef-fect will definitely be passed on. Currently, commercial search engines such as Google, Bing and Baidu utilize text-based retrieval techniques in their service and have made magnificent success from them. However, the text-based image search results are still far away from users’expectation, and have plenty opportunities to be improved. Re-cently, some techniques are proposed to enhance the text-based search result using content-based techniques, including image search reranking, image search result sum-marization and thumbnailing. In this dissertation, we propose several solutions based on object semantics to enhance the quality of text-based image search results, including image search reranking, image search result summariztion and thumbnail generation. The main contributions of this dissertation are summarized as follows:1) A retrieval-oriented object semantics extraction method, optimized on ob-ject semantic mining for object queries. Since the key words for objects occupy quite a bit of search queries, how to fully explore the relationship between the query and the objects in the result images has become a crucial factor for the search quality. Moti-vated by that, this dissertation proposes to mine the underlying object categories using query object discovery with ROI-based image representation and construct a query-specific semantic space using query expansion. The constructed object vocabulary offers firm support to the following search result enhancing techniques.2) A novel image search reranking method based on object-based language model, which substantially improves the search relevance of text-based image search. Although the researchers have proposed several effective relevance assump-tions, their performances are limited due to background noise. Motivated by that, this dissertation suggests to re-formulate the reranking process though object-based lan-guage modeling and predicts the image relevance via a risk-minimization framework. Since language models for text-based queries cannot be directly predicted, this dis-sertation re-implements the classical relevance assumptions in the object level, which predict the query language model by mining the object-based visual content and text-based search ranking. In order to integrate the advantages of different assumptions, this dissertation proposes to train a combination model from human labeling via a su-pervised reranking framework.3) An image search result summarization method considering multiple object-based summarization factors, which outperforms conventional methods on object queries. Conventional works propose that the quality of image search result sum-marization depends on multiple factors such as relevance, diversity and attractiveness. Through the study of user demand on object queries, we conclude that the diversity of the search result for object queries has multiple levels. To capture such characteristics, this dissertation proposes an object category selection method based on non-maximum suppression. This dissertation also assumes that the attractiveness of object images de-pends on the object location in the image and its size. We design an evaluation method to realize such criterion and integrate the above two factors into a probability-relaxed optimization framework.4) A query-dependent thumbnail generation method, which improves the foreground localization accuracy and grantee query relevance when multiple ob-jects are presented. Since the challenge is to optimize localization accuracy while considering query-relevance, this dissertation proposes to detect correct foreground object region using the object vocabulary, which fulfils this objective using object se-mantics. In order to produce high-quality thumbnails based on the localization result, we propose a complete solution to generate satisfactory cropping boundaries, including bounding box expansion, refinement and missing background inpainting.Since enhancing image search using the semantic information extracted from it-self has become a trend in the research field, this dissertation serves as a first attempt in designing specific approaches for object query processing. This attempt will inspire following works for other kinds of queries.