Research On Visual Relationship Detection In Natural Scene

In natural scene understanding,Visual Relationship Detection(VRD)is an important task,which aims to detect objects in images and identify the relationships between objects.There are many objects and relationships in images.However,from a human perspective,we usually pay more attention to some important objects or relationships.Existing methods do not distinguish the importance of objects and relationships,resulting in poor detection results in complex real images.In addition,because the distribution of visual relationships is long-tailed,detection model needs the ability of few-shot learning even zero-shot learning.To solve the above problems,we propose an attention-based multimodal visual relationship detection model AVR.AVR detects visual relationships by fusing visual,spatial and semantic features,and recognizes the importance of objects and relationships based on attention mechanism.The greater the attention to the object or relationship,the more important it is in the image.The attention of objects and relationships will be used to enhance the visual relationship detection.In addition,to fine-tune the relationship prediction of AVR to improve accuracy,we propose a random-walk based algorithm to obtain the priori of visual relationship.To mine the association and dependency between the relationship prediction,the attention of objects and relationships,and the relationship priori,we propose a heuristic learning framework based on genetic programming algorithm,which is used to automatically evolve heuristic rules to combine these variables.The best evolved heuristic is applied to further improve the performance of visual relationship detection.Comprehensive experiments on several real-world datasets are conducted to demonstrate the effectiveness of the proposed algorithms.The results show that the proposed algorithms outperform state-of-the-art visual relationship detection methods significantly in terms of recall.Meanwhile,the best evolved heuristic can further improve the performance of relationship detection.