The Recognition And Retrieval Of Hand-drawn Sketches

With the rapid development of digital era, everyone could become the publisher of multimedia data. Massive amount of data are being uploaded to the servers all over the world at all times, and image retrieval is a necessary and desirable part of search engines. Content-based image retrieval (CBIR) uses images as the query data, searches and returns images from the database, in which the images are visually or semantically relevant to the query images. Compared to text-based image retrieval, CBIR is more convenient and has a higher chance and opportunity to return the results that are more useful for the users since images contains much more information than texts. CBIR is one of the main research area in multimedia retrieval.The popularity of touch-screen devices brought great changes in the mode of interactions. Sketch-based image retrieval (SBIR) is a branch of CBIR, which uses hand-drawn sketches as query data. There is no need for users to look for specific images as queries, they can draw freely on their touch-screen devices to representation what they want to search. The boundary of an object is an important part of human's subjective perception, which is a highly abstract description of the object. SBIR is a convenient way for searching and is more in line with the human's intuition.Hand-drawn sketches are highly abstract, and they may contain non-rigid deformations, which makes it more difficult for image retrieval. The hand-drawn sketches may have different rotation, scaling and translation (RST) with the real images. In this paper, to overcome this issue and considering the non-rigid deformations, we proposed an SBIR method which is RST-invariant. In order to invariant to scaling and translation, we extract the main parts of both hand-drawn sketches and real-world images by combining the salient maps and the boundary maps. The main parts are cropped and resized to a same size for further retrieval process. By compressing the feature in a local patch and searching for neighborhood patches, we eliminated the influence of non-rigid deformations. To achieve rotation-invariance, we divided the images into sub-regions and matched the sub-regions with minimum-weight perfect matching. The experiments on three datasets shows that our method outperforms the state-of-the-arts in both natural images and product images.Some hand-drawn sketches may be too abstract to search only based on the similarity of visual features. The semantic information could be extracted from the hand-drawn sketches as the auxiliary information for image retrieval. Aiming at this phenomenon, in this paper, we extracted the low-level Gabor features from sketches and trained a two-layer sparse auto-encoder to extract mid-level features from low-level features. With the help of calculating the inter-class similarities, we trained two loops of classifiers to recognition the class of hand-drawn sketches. Then we combined the visual similarities and the latent semantics in the sketches to enhance the retrieval performance. The experiment on two datasets shows that the proposed method has a better classification ability, and enhanced the retrieval performance on the dataset with tags.