Research On Scene Graph Generation Method With Relation Feature Enhancement

Scene graph generation is designed to enable machines to recognize objects in images and determine their relationships,just as humans do.The scene graph is widely used in high-level image understanding fields such as image subtitle generation,image retrieval,and visual question and answer,which have important research value and practical significance.At present,there are still two problems in the task of scene graph generation.On the one hand,most of the existing scene graph generation methods only use the context between objects to represent the visual relationship,ignoring the information interaction between the relationships and the relative position information between objects.As a result,the lack of context between the relationships and the relative position information results in inaccurate model prediction.On the other hand,due to the long tail distribution of the dataset in the scene graph generation task,the trained model is not able to perform well in extracting uncommon relationship features,which results in the model preferring to predict common relationship types,while for uncommon relationship types,the model is not effective.To solve these problems,this thesis mainly does the following work:(1)A feature enhanced scene graph generation method based on hierarchical context and relative location is proposed.This method utilizes the Transformer structure to interact with the context information of objects and relationships,extracts the relative location information between objects before interacting with the context information of relationships,and integrates it into the relational feature representation.Through the interaction of object and relational context information and the integration of relative location information,the model’s representation of relational features is enhanced,thereby improving the model’s reasoning ability.The method was tested on a VG dataset,and the results of comparative and ablation experiments were analyzed to verify that the proposed method can improve the performance of scene graph generation.(2)A method for generating scene graphs enhancement with relational features based on external memory is proposed.This method designs an external memory module to store the relationship features from different images to transfer the relationship information across images.By reading the related relationship features in the memory module,we enrich and enhance the representation of low-frequency relationship features,which can improve the prediction ability of the model for low-frequency relationship categories.The model also experimented on a VG dataset.By analyzing the experimental results,it is verified that the method can improve the prediction ability of the model for uncommon relationships.(3)A visual relationship detection system based on deep learning is designed and implemented.The system integrates the scene graph generation methods proposed in this thesis and the mainstream scene graph generation methods,which can achieve the functions of target detection,visual relationship detection,and so on.In summary,this thesis puts forward some improved solutions to the problems in the task of scene graph generation,and designs and implements a visual relationship detection system that contains the methods presented in this thesis,which can help researchers better understand the process of visual relationship detection.