Research On Sketch Retrieval Method Based On Deep Learning

Sketch-based image retrieve(SBIR),allowing to retrieve nature images from ambiguous and abstract sketch queries,compared to TBIR and CBIR techniques heavily relying on either providing a textual description or a similar search target,freehand sketches are more intuitive and able to convey rich information precisely for humans since “an image is worth a thousand words”.How to reduce the domain gap between the sketch and the natural image is a key issue in SBIR research.The traditional method extracts manual features to complete the approximate conversion between the sketch domain and the image domain,but this kind of method can not fit the contents of the two domains,so it can not effectively reduce the domain gap.Existing deep learning methods attempt to narrow the distance between domains through single-sided domain conversion methods,such as converting natural images into sketches or converting sketches into natural images.However,the single-sided domain conversion method has certain limitations and still cannot fundamentally reduce the domain gap.Aiming at the above key issue,this article proposes a deep sketch retrieval method called Strokharmon.This method focuses on realizing the bilateral domain conversion of the images,which greatly reduces the domain gap between the natural image domain and the sketch domain,and improves the sketch retrieval accuracy.The main research work is as follows:1.First of all,this article summarizes the existing sketch retrieval methods based on deep learning and divides the existing sketch retrieval research content based on deep learning into four categories: coarse-grained retrieval,fine-grained retrieval,deep hash technology,and category generalization.The main research issues are summarized,commented on,and comparatively studied.2.In order to solve the problem of the large gap between the natural image domain and the sketch domain in sketch retrieval,a deep learning method for sketch retrieval called Strokharmon is proposed.This method is mainly composed of two networks,one is Strokharmon-GAN,which is a stroke harmonization generative adversarial network for the realization of bilateral domain conversion,and the other is the network model Strokharmon-Res Net for deep feature extraction in the stroke domain.The Strokharmon bilateral domain conversion sketch retrieval method realizes the alignment of natural images and sketches in the same intermediate image domain,completes the conversion of data from different domains to data in the same domain,and realizes sketch retrieval through the method of same domain feature mapping.3.In order to verify the effectiveness of the Strokharmon method proposed in this article,this paper has carried out two parts of experimental verification:(1)The validity of the Strokharmon-GAN domain conversion model for the harmonization of strokes proposed in this paper is verified.The experimental results show that the domain conversion method proposed in this paper is better than the existing domain conversion models for sketches;(2)It is verified the retrieval performance of the Strokharmon sketch retrieval method proposed in this paper is comprehensively evaluated on two representatives public fine-grained SBIR data sets.The experimental results show that our method obtains 85.6% and 96.3% retrieval accuracy on QUML-Shoe V2 and QUML-Chair V1,respectively,which are 4.9% and 0.4% higher than the current latest method,and obtain better retrieval performance.