Local Semantic-Aware Fine-Grained Sketch-Based Image Retrieval

With the rapid increase of image content on the Internet,the demand for image retrieval is increasing.Currently,the main image retrieval methods include text-based image retrieval and content-based image retrieval.However,text-based image retrieval is difficult to retrieve complex images and requires a lot of manpower for annotation.Contentbased image retrieval also faces difficulties in acquiring input images.Different from text and natural images,hand-drawn sketches are rich and easy to obtain,making sketch-based image retrieval a more ideal image retrieval method.Sketch-based image retrieval can be divided into coarsegrained and fine-grained,among which fine-grained sketch-based image retrieval can better capture user needs and is more difficult to retrieve.For this task,researchers mainly adopt the metrics-based learning method,that is,first extract the global features,and then map the features to a unified feature space through optimization.However,due to the use of global features,the model can only perceive the global semantics,and has difficulty in perceiving the more important local semantics.Therefore,this paper aims to study local semantic-aware fine-grained sketch-based image retrieval.In order to make the model perceive local semantics,this paper innovatively proposes to use local features for fine-grained sketch-based image retrieval,and proposes Dynamic Local Alignment Network(DLANet).DLA-Net contains two module:local feature extractor and dynamic alignment module.The local feature extractor uses the mid-level feature map output from ResNet50 and local L2 normalization to obtain local features for retrieval,and the dynamic alignment module finds the matching relationship between features through cross-modal interaction,which is proposed to solve the spatial misalignment problem.Experiments show that the retrieval accuracy of DLA-Net significantly outperforms existing methods on commonly used datasets,and its retrieval accuracy outperforms that of humans for the first time on all datasets.For retrieval tasks,retrieval time is also an important metric to be considered.Although local features greatly improve the retrieval accuracy of DLA-Net,they also reduce the retrieval speed.Therefore,this paper proposes a method to improve the retrieval efficiency of local features,which includes a self-interaction background feature elimination module and a feature dimension reduction module.The self-interaction background feature elimination module locates and eliminates background local features using a priori information of the former region and background region in the image.The feature dimensionality reduction module uses 1 × 1 convolution to reduce the feature dimension and uses a two-stage strategy for training.Experiments show that these two modules can significantly improve the retrieval efficiency of DLA-Net while ensuring its retrieval accuracy.Finally,this paper also develops a fine-grained sketch-based image retrieval system based on the algorithm proposed above,through which the application of fine-grained sketch-based image retrieval is explored.