Representations, feature extraction, matching and relevance feedback for sketch retrieval

Sketching is a natural way of input that provides an effective means of illustration. A sketch consists of multiple strokes that can be captured by pen-based devices to be stored in a database for future retrieval. Linear browsing is not feasible when the number of sketches in the database becomes large. Not only do we need a tool in retrieving sketches, but also we need an efficient system that can provide retrieval result with high recall and high precision. In this thesis, we propose a novel approach that includes several aspects in improving the sketch retrieval performance. Given a sketch, we propose to have multiform representations of this sketch in order to find at least one consistent representation when it is compared against other similar sketches under user variations. We then propose to perform coarse-to-fine feature extraction in order to capture the characteristics of the sketch at various levels. We build a classifier in shade region detection for hand-drawn sketches and for images. When two sketches are compared, we propose to have global and local matching that computes the similarity not only based on the shape information, but also based on other criteria such as spatial relations and the structures. In addition, we propose to extend traditional single component relevance feedback to multiple component relevance feedback in order to refine the retrieval result based on the user feedback. Finally, we will show our approaches for solving the partial matching problem. In the last part of this thesis we will show several prototypes that we have been implementing in order to demonstrate how sketch retrieval can be applied in real applications.