Research And Implementation Of Scene Graph Generation Algorithm Based On Attention Mechanism

Scene graph,as an important semantic representation,can enhance a visual model reasoning ability,i.e.image caption and visual question answer.Correlation generation algorithm has become one of the focus in the field of computer vision due to its essential research significance and potential applications.Taking scene graph as research object,this paper proposes three models for scene graph generation from the perspectives of optimizing information passing and introducing non-visual information.The main works of this paper are summarized as following.This paper proposes a Dual Attention Scene Graph Generation Network(DANet)to utilize dual attention mechanism to guides message passing among nodes efficiently towards the unbalanced issue in message passing,where external attention mechanism is used to determine the weight of different nodes and internal attention mechanism is used to modulate the transmitted message.On VRD,comparing with F-Net,the proposed DANet improves performance by 2.07% and 2.06% on Phr Det and SGGen respectively.This paper proposes a Hierarchical Message-Constraint Network for Scene Graph Generation(HMCNet).HMCNet consists of three core parts: hierarchical message passing module is used to refine object and subgraph features in message passing layer;Spatial-Constraint Relation module is used to guide the fusion of node features in spatial feature layer;statistical co-occurrence correction module is used to regularize relationship prediction by statistics information of dataset in classification layer.Ablation experiments on VRD are carried out to show the effectiveness of each parts of the proposed model.Comparing the mainstream one-stage models on VRD,VG-MSDN and VG-DR-NET,the proposed model achieves the state-of-the-art.This paper proposes a Lite Scene Graph Generation Model Based on KnowledgeEmbedded Routing(KLNet).KLNet can take full use of semantic information to regulate the fusion between nodes in relationship prediction module.Meanwhile,reasoning ability of the proposed model can be enhanced by adding the number of nodes in relationship prediction graph.To fast the training,the proposed model utilize attention prune to reduce the computation of message passing.The experiments on VG-IMP show that the proposed model outperforms KERN by 2.9% and 1.9% on Pred Cls and SGCls with 20% improvement of training speed,which demonstrates the effectiveness and practicability.