Image Retrieval And Implementation Based On Object Relationship Structure

Object relationship detection in images can mine more fine-grained features and provide structured image content representation to improve the performance of image retrieval in complex scenes.In this paper,object relationship prediction in images and its application in the field of image retrieval are studied.The visual relationship prediction model is constructed by using features extracted from images,and a knowledge-enhanced approach is used to deal with the long-tailed distribution problem.According to the scene graph,which is composed of object relationships,hard and soft matching methods are proposed to improve the image retrieval ability in complex scenes.The specific work includes:(1)Object feature representation and priori knowledge representation fusion learning: a method of modeling object features and prior knowledge jointly is proposed to predict the relationship between objects.Object detection model is used to get the location of object in images,and the visual relationship prediction model is constructed by the visual features of objects,the location features between objects and the category features.Through symbol representation learning,vectors of objects and relationships are obtained and to construct the semantic relevance of the object relationships using a map function.The distance between similar types of relations is reduced by prior knowledge representation learning.It can improve the accuracy of the prediction and solve the zero-shot or few-shot problem caused by the long tail distribution of object relationships.(2)Image retrieval based on scene graph: a method of image retrieval based on scene graph,which is composed of object relationships,is proposed to solve the problem of image retrieval in complex scenes.This paper provides two retrieval methods: image as query and nature language as query.Two matching methods based on hard matching and soft matching are proposed to improve the semantic matching ability of scene graph.(3)Image retrieval system prototype implementation and experimental verification: Object relationship prediction model is deployed in the retrieval application to implement the system prototype of image retrieval.The accuracy of object relationship prediction model and its effectiveness in image retrieval are verified on VRD and Visual Genome datasets.Experiments show that the visual relationship prediction model constructed by object feature representation can capture the relationship between objects.The accuracy of visual model prediction is significantly improved by using prior knowledge enhancement and the recall values on VRD and Visual Genome datasets are increased by about 21.3% and 16.8%,respectively.Compared with extracting the whole image features for retrieval,the median rank value of using scene graph composed of objects relationships has been improved from 20 to 6 in the field of image retrieval.