Image Representation Models Based On Local Features: Theory And Practise

Image classification and retrieval have been core problems in computer vision, pattern recognition and machine learning. Image representation based on local features is the most popular approach in image classification and retrieval. However, due to the wellknown semantic gap and the limitations of local features in representing high-level visual concepts, conventional image representation models often su?er from a lot of shortcomings, including the sensitiveness to small noises, the lack of spatial structure information in feature encoding, the di?culty in capturing regions-of-interest in specified classification and/or retrieval problems, etc.This thesis presents extensive research e?orts to combat the above issues, providing insightful improvements and discussions to two types of image representation models based on local features, namely the Bag-of-Visual-Words(BoVW) model and the deep Convolutional Neural Network(CNN). Starting from real-world applications, we abstract several important scientific problems and suggest novel solutions. We partition image representation into several modules and explore each one of them in depth, including feature extraction, feature encoding, feature summarization, post-processing, etc. Based on modular research, we propose a pioneering unified model to deal with both image classification and retrieval problems. Finally, we suggest two challenging problems in computer vision, and provide primary yet innovative approaches to deal with them.The main innovations of this thesis are summarized in the following six aspects.? We propose an algorithm for local feature enhancement. Based on the observation in real-world classification and retrieval problems, we demonstrate the importance of reversal invariance of local features, and then design a straightforward solution.? We propose an algorithm which applies spatial information to enhance feature encoding. With the construction of “geometric visual phrases”, we embed more powerful descriptive power into the encoded features.? We propose two spatial matching algorithms to cope with two specified classification problems, i.e., fine-grained object recognition and scene recognition, and improve image representation quality.? We propose two post-processing algorithms for image retrieval and large-scale Web image search. Based on the graph-based data mining algorithms and the random walk theory, both the precision and recall of image retrieval are significantly improved.? We propose a unified model for both image classification and retrieval, which is on the basis of powerful regional features and robust computation of the image-to-class distance. To our knowledge, it is a very first trial towards unifying these two problems, which also achieves state-of-the-art performance.? We propose two challenging research topics in computer vision, and provide elementary efforts based on state-of-the-art techniques and innovative organization structures. These works might pave a new way to future researches in the computer vision community.One major contribution of this thesis lies in the powerful generalization ability of the proposed methods. Most of them could be applied to various problems, i.e., not limited to the evaluated cases, and produce consistent improvements. Our research provides several new clues for researchers on the related research fields. The proposed interesting yet challenging problems also lay the foundation of our future works.