Sketch Based Image Retrieval On Deep Learning

With the rapid development of the internet age,the number of digital image data has increased dramatically.Here is an urgent need for a huge number of digital image data in the image retrieval task.The traditional image retrieval tasks include two methods,Text-Based Image Retrieval(TBIR)and Content-Based Image Retrieval(CBIR).Because of the way to rely solely on keywords for retrieval,TextBased Image Retrieval often cannot clearly express people's search needs.In contentbased image retrieval task,it often fail to get the correct retrieval results because people need to provide query image.Because there are the disadvantages of the above search methods,and the universal popularity of touch devices,the image retrieval method based on freehand sketches can be proposed and developed rapidly.Sketch based image retrieval(SBIR)aims at measuring the similarities between sketches and real images.In the task of sketching image retrieval,retrieval tasks face enormous challenges because of the highly abstract and structurally diverse nature of hand-drawn sketches.Most existing approaches tackle this problem via projecting feature representation of these two domains into the common feature subspace without effectively utilizing the structure of the query sketch.We argue that the feature representation of real images should be developed conditioned on the structure of query sketch.Therefore,inspired by the strategy of repeatedly viewing local images when comparing images with the human,and combining the feature of convolution neural network to be more robust,this paper proposes a new framework of sketch image retrieval based on deep learning.To that end,in this paper,we introduce a two-stage framework for the query sketch-aware feature representation learning.Our stage-1 network is the Flexible Latent Alignment Generator(FLAG)which is composed of recurrent neural network.FLAG is able to dynamically align the sketch and real images with a region comparator,and provide the initial candidate regions for the next stage.Our stage-2 aims to learn the discriminative feature representation with a latent semantic co-attention mechanism.The spatial co-attention network can adaptively develop the feature representation of real images conditioned on the query sketch.Extensive experiments conducted on Sketchy dataset shows that our proposed framework achieves impressive retrieval performance,especially for sketch retrieval based on zero-shot learning.