| With the development of science and technology and the increase of the amount of data,various tasks in the field of computer science can be solved by data-driven methods.Among them,the processing method for single media(modality)has become mature in some tasks,while the joint application of multi-modal data with certain semantic correlation for different modalities is still in the immature development stage.Visual question answering is a pioneer work in this kind of task.Its task purpose is to give reasonable answers according to pictures and question text content.The task requires the model to have the ability to jointly understand the semantic information of image and text,and get the final output through reasoning,fusion and other different ways.In the research work of Visual Question Answering(VQA)task in recent years,different methods of mapping objects in the images and supplemented by graph neural networks emerge one after another.Benefiting from the gradual maturity of the scene graph generation task proposed in the Visual Genome dataset,and the thinking of the spatial position from the object detector output,the method of applying scene graph(Scene Graph)and spatial graph has gradually become one of the mainstreams of graph methods in VQA task.Most of these methods focus on graph construction and the information flow in graphs.By constructing different graph structures,different prior knowledge is incorporated into the model,and these strong priors are used to increase the performance and generalization ability of the model.It is important to design a more detailed and more specific graph neural network to make the information flow within the model more reasonable and enhance the interpretation and representation capabilities of the model.However,in the current existing methods,the mainstream methods do not well separate different information fusion stages,resulting in low information refining ability of single-modality itself,or relatively designing rough multi-modal feature fusion.In the process of applying graph network information refinement,the fusion mechanism for information between different semantic level of single modal is relatively simple,which need to be designed more elaborately.In view of the above problems in the previous work,this paper designs a more refined graph neural network structure,removes more redundant information in the refining process of single-modal information,and designs a new multi-graph fusion method.In different graphs neural networks,appropriate fusion methods are selected for different problems to aggregate multi-graph features.The specific work is as follows:(1)In order to make our model have the ability to extract semantic information from different dimensions,and at the same time not to introduce strong prior knowledge,this paper designs three different graph structures.The three graph structures are:spatial graph,semantic graph and latent relation graph.These three graph structures are assigned with different weights according to different problems,and the final analysis confirms that the first two graph structures can improve the performance of the latent relation graph,that is,the prior knowledge that the fully connected graph lacks,and at the same time benefiting from the flexibility of the fully connected graph,the three form a relatively complementary information relationship,which makes the model design more reasonable and enhances the model performance.(2)Aiming at the potential redundancy in the information flow in graphs and the information loss in the process of graph neural network transmission,this paper designs a new graph neural network structure by retaining historical graph structure information and de-redundancy operations in any single-layer of the graph neural networks to ensure the consistency of different layers in the graph neural network structures.This paper also tried to remove the influence of some redundant nodes and edges in a single information flow process.In ablation studies,it is verified that these methods can improve the overall performance of the model.(3)Aiming at the feature fusion problem in the multi-graph structure,a multi-graph fusion method based on pseudo-labels is designed,and auxiliary loss functions ate designed to ensure the stability and robustness of the training of the module,which greatly improves the overall performance of the model.Numerically shows the primary and secondary relationship between various mapping methods,which provides a certain experimental basis for subsequent work. |
