Research Of Deep Learning-based Sketch-based Image Retrieval

Fine-grained Sketch-based Image Retrieval(FG-SBIR)is designed to solve the problems of instance-level sketch and natural image matching.Currently,the key challenge of FG-SBIR are:(1)The sketches are very abstract,and the draftsman's skill varies greatly,resulting in very different drawing results.(2)Sketch annotation requires a lot of time and effort,so the sketch data set is still missing,and the number of samples in several open sketch data sets is small.(3)There is large difference between the sketch and the natural image.However,the FG-SBIR model not only needs to learn the characteristics of different modes,but also maps them to the same space for distance calculation to realize cross-modal retrieval.(4)FG-SBIR needs to solve the fine-grained matching problem,which requires the model to have a high recognition ability and be able to distinguish different instance pictures.In this thesis,we conduct in-depth research on the related issues of FG-SBIR.In the aspect of feature learning,the optimal feature extraction model for sketch is found by the method of sketch recognition.For the problem of large difference in sketch and small amount of data,we adopt a sketch data enhancement method based on shaping transformation for data expansion,which achieves effective improvement in the sketch recognition task.Combining the visual information and temporal information of the sketch,we explore a variety of different sketch recognition methods based on Convolutional Neural Networks(CNN),coordinate sequences and stroke-based sequences,and analyze the recognition performances of different methods.In terms of image retrieval,in order to improve the recognition ability of the model,we propose an enhanced hard triplet construction method,and through a large number of comparison experiments,the method can significantly improve the retrieval accuracy.In addition,we analyze the impact of category information and weight sharing methods on cross-modal retrieval results,and explore a better solution for FG-SBIR.