Research On Freehand Sketch Retrieval Technology Based On Local Detail Feature Mining

The popularity of mobile devices has made freehand sketch a simple and efficient way to query.For image retrieval,using freehand sketches to express visual cues and query intent is more convenient than using natural images and more efficient than using text.In this case,freehand sketch-based image retrieval shows great potential in the practical application of computer vision.Due to the strong feature representation ability and translational invariance of convolutional neural networks,the freehand sketch retrieval methods based on deep feature have made great progress in recent years.However,freehand sketch retrieval is still subject to certain challenges,including the lack of sketch samples of the same category,insufficient extraction of local detail features,and difficulty in cross-domain matching of sketches and images.Focusing on above problems,this dissertation studies from the perspectives of sketch data augmentation,discriminant local part mining,fine-grained feature extraction and crossdomain feature matching.To this end,this dissertation aims to improve the shortcomings of the existing methods,so as to further improve the effect of freehand sketch retrieval.The specific research content and main contributions are as follows:1.An augmentation method based on part segmentation is proposed to model the abstraction and diversity of freehand sketches.Firstly,using the structural information of strokes and the correlation between different strokes,the semantic labels are assigned to the strokes with the help of the graph convolutional network as well as the sketch divided into parts according to the semantic labels.Then,two sketch data augmentation strategies,i.e.,random deletion of parts and random exchange of parts,are proposed based on the split parts.The random deletion of parts can generate abstract new sketches by deleting some parts with less semantic information,while random exchange of parts is to generate some diversity of new sketches of the same category by randomly selecting a certain number of parts with the same semantic label from two sketches of the same category for exchange.In addition,these two augmentation strategies can be superimposed with shape deformation-based augmentation methods to enhance the richness of sketch data.Finally,the experimental results prove the effectiveness of the augmentation method that can effectively improve the performance of the basic model and is endowed with certain practical value in freehand sketch retrieval tasks.2.In order to obtain a more discriminant cross-domain feature representation,a sketch retrieval method targeting at discriminant local part mining is proposed.Firstly,a recurrent multi-scale network is designed to gradually find the most discriminating local part by focusing first globally and then locally.In this network,local parts are mined automatically using the regional positioning network,the style diversity of local parts is solved by adding part-aware module,and multi-scale joint feature representation is generated by feature fusion operation.Then,in order to learn the cross-domain feature representation of sketch retrieval,the recurrent multi-scale network is extended to a three-channel format that integrates sketch,edge map and RGB image branches,and is alternately optimized by weighted regularization triplet loss,classification loss,and ranking loss.Finally,the experimental results on the five sketch retrieval datasets demonstrate that the proposed method effectively improves the discriminant power of the network,thereby achieving a better sketch retrieval effect.3.A fine-grained sketch retrieval method targeting at fine-grained feature mining is proposed to fully extract fine-grained information from local features.Firstly,a three-way enhanced part-aware network is constructed by introducing a mixed high-order attention and local enhancement module into the backbone network.This very network can generate various high-order attention feature maps and enhance the representation of useful detailed features.Then,emphasizing inter-class separation and intra-class compactness,an improved triplet loss is designed to establish cross-domain bidirectional constraints,which is used to optimize the network together with classification loss and adversarial loss.Finally,experiments on finegrained popular datasets verify the effectiveness and superiority of the proposed method in finegrained retrieval.Additionally,this method improves the ability of the network to obtain discriminative fine-grained information,and can be used as a reference for other fine-grained research.4.A fine-grained sketch retrieval method based on cross-domain local feature interaction is proposed to solve the problem of the difficult alignment of cross-domain local features caused by spatial dislocation.This method starts with the correlation of local features between sketches and images,and establishes the correct matching relation through the interaction of crossdomain local features to achieve feature alignment.In order to achieve fine-grained level matching,the local feature extractor is used to extract fine-grained features,then the local selfattention is used to capture long-distance interaction of local pixels,and finally the crossdomain local interaction module is to align local features.A large number of comparative experiments on five public datasets demonstrate the significant fine-grained retrieval effect of this method.Meanwhile,cross-category generalization experiments also prove the crossdomain generalization ability of the method.Overall,this study provides a reference for the practical promotion of subsequent sketch retrieval technology.