Image Representation For Content-Based Large Scale Image Retrieval

Existing large scale image retrieval system has been significantly advanced by the introduction of robust local features and the bag-of-words image representation. Bag-of-words representation, however, has significant amount of information loss in the pipeline of converting an image into the visual words. Also, the general image features used in the bag-of-words representation can't well capture the special properties of some important categories of objects in the image. We propose a novel image representation framework consists of"visual word","visual phrase"and"visual theme"to better handle these problems: 1) by encoding more information of the original image feature, we generate a more discriminative"strong visual word"codebook; 2) we combine multiple"visual words"to generate the more discriminative"visual phrase", in which robust geometric constraints can be efficiently enforced; 3) we categorize images into different"visual themes", and design different image features accordingly to exploit their special properties. The main content and innovations in this dissertation are as follows:1. We propose a"multi-sample, multi-tree"based"strong visual word"image representation. Based on the analysis of the information loss in converting an image into a bag of"visual words", we propose an approach to sample multiple image patches at each image feature point/region to encode more complementary information about the underlying image feature. We also propose a"multi-tree"quantization approach to efficiently generate a visual vocabulary with finer feature space partitions, therefore reduce the quantization error.2. We propose a"visual phrase"image representation with bundled features. We develop a novel scheme where image features are bundled into stable local groups using a stable region detector. And then robust geometric constraints can be enforced in these groups. These bundled features are not only repeatable, but also much more discriminative than a single feature.3. We develop a novel image representation for face image retrieval, in which we propose an"identity-based"quantization scheme and a"multi-reference"re-ranking scheme to cope with pose and expression changes. We discuss the importance of"visual theme"in designing features for image retrieval, then focus on the face image retrieval domain, which is an important specific"visual theme"in web images. We develop a new face representation using both local and global features, where local features are firstly used to efficiently achieve candidate images with high recall, and then global features are applied to re-rank the candidates for accuracy.To summarize, we propose a novel image representation framework, and provide detailed solutions for each component in the framework. We apply these solutions on the content-based large scale image retrieval to prove their reasonableness and effectiveness.