Research On Key Technologies Of Sketch Based Image Retrieval

Freehand sketch is a visual media form that can express human ideas or concepts clearly with a few simple strokes.The image retrieval of freehand sketch refers to the automatic process that the retrieval system returns the same kind of image or the specific precise matched image after feature extraction and matching under the given query sketch.In recent years,with the popular-ization and application of modern electronic touch screen and pressure sensor devices,researchers began to pay more attention to how much the computer can do when recognizing sketches.Therefore,SBIR and FG-SBIR have become new research hotspots.This paper focuses on two major difficulties of the cross-media informa-tion retrieval task of freehand sketches and rdal images and images are heterogeneous media,with data non-homology,it belongs to cross-media re-trieval;(2)sketches are mainly composed by simple lines,lack of texture and low-level features,sketch retrieval needs more effective feature extraction and representation method.In order to reduce the inter-domain difference in cross-media retrieval,this paper studies the natural image sketching method,which converts the image into sketch-style result.For the research of sketch feature extraction and representation,this paper studies the deep model of sketch se-mantic attributes and the corresponding retrieval algorithm.The main work and innovations of this paper are as follows.1.Propose a photo-to-sketch transformation algorithm based on saliency detectionThe commonly used method of narrowing the domain gap difference is generally to take pre-processing operation of extracting edges of natural images.However,this pre-processing step can not make a perfect match between the individualized hand-drawn sketches and the natural images frequently.There-fore,this paper proposes an image-to-sketch transformation model based on saliency detection.In order to highlight the foreground region of interest while removing the unrelated background of images,we firstly propose an improved algorithm based on Markov model.The texture details in natural images are then further processed by proposing a specific Gabor filter to suppress redun-dant small edges due to the detail texture.The experimental results of SBIR on Flickrl5k database show that the proposed algorithm can not only generate more sketch-style images,but also improve the SBIR performance.Especially,it improves the accuracy of about 15%on image categories with complex back-ground,which proves the effectiveness of the algorithm.2.Propose a FHS-GAN model based on deep GANThe results of photo-to-sketch transformation model of saliency detection can indeed improve the accuracy of SBIR.However,it is not effective in the more difficult FG-SBIR task,therefore,we propose a FHS-GAN model based on GAN that can automatically generate sketches.A new objective function is designed to improve the training stability,while the gradient penalty term is introduced to strengthen the constraint of the adversarial loss so that the model can be further optimized.The generated results on Sketchy database show that the FHS-GAN model can obtain sketches that are closer to the form of freehand sketches.In addition,in order to objectively evaluate the stability and domain adaptability of FHS-GAN model,an improved similarity model based on Faster R-CNN is proposed.This model can not only explicitly and obj ectively measure the quality of generated sketches,but also be used as a complete hand-drawn sketch recognition model.A large number of experiments are carried out on the cross dataset of Sketchy and Flickr 15k by using this metric model.The experimental results show that F HS-GAN can produce higher quality and more realistic sketch results,which is more suitable for FG-SBIR task.3.Propose a deep deformable model for the attribute prediction of freehand sketchesFor the task of SBIR,most of the methods are to extract the deep features of freehand sketches for subsequent feature matching and retrieval.But for the task of FG-SBIR,deep features are not sensitive to discriminate differences be-tween instances.Therefore,we study the semantic attributes of sketches and construct a deep deformable network for sketch attributes detection.Firstly,a deformable CNN unit is constructed to obtain a variable perception field to better deal with the variability of freehand sketch.Then,according to the deter-ministic characteristics of sketch attributes' spatial location,a network structure combined with attention mechanism is proposed to predict sketch attributes.In addition,the experimental results on the QMUL-Shoe,QMUL-Chair and self-built clothes database with sketch attribute annotations show the effectiveness of the deep deformable model proposed for sketch attribute prediction task.4.Propose a deep deformable triplet model for FG-SBIRIn order to improve the retrieval accuracy of FG-SBIR,we propose a deep deformable triplet model by introducing high-level semantic attribute features to narrow the semantic gap in human vision.The model firstly uses the attribute prediction network and expands it into three branches of triplet structure.Then,the results of FHS-GAN model are used to narrow the data difference between sketch and image.Finally,the attribute features obtained by the attribute predic-tion model are fused to further improve the distinguishability of features.We conduct verification experiments on the self-built,QMUL-Shoe and QMUL-Chair database.The experimental results show that the combination of deep and high-level semantic attribute features can improve the distinguishability and generalization ability of sketch features,and improve the retrieval accuracy of FG-SBIR with a maximum of about 4%percent.In addition,competitive re-trieval results have been obtained on large-scale Sketchy dataset,which proved the validity of the proposed model.