| The information carriers in the internet exhibit diverse forms,which leads to a high demand for cross-modal processing techniques.As an important task in the cross-modal field,Visual Question Answering(VQA)aims to enable computers to answer questions posed by humans through analyzing images.VQA requires computer models to understand the semantic information of both image and question text,while eliminating semantic differences between the two modalities and inferring the answer,making it highly challenging with broad application prospects.To aid in the inference,external knowledge graphs are introduced to enhance the representational capability of image features.This thesis addresses the insufficiency of cross-modal interaction and the inadequate reasoning ability caused by a large amount of noise from external knowledge by exploring two directions: enhancing the semantic interaction between images and text and introducing external common-sense features.Therefore,this thesis mainly studies the following aspects based on deep semantic fusion:(1)Research on Feature Extraction and Deep Fusion Method based on Modal Joint Interaction for Image-Text.This method addresses the issues of insufficient interaction between image features and textual question features in existing methods,resulting in polarized reasoning results.By introducing a mechanism of modal joint interaction,this method achieves bidirectional guidance between image and question features,thereby enhancing the interactive capabilities of the model.Additionally,by using a residual-based deep stacking fusion mechanism,it further enhances information sharing in the crossmodal semantic space.Finally,through the design of multiple experiments,this paper demonstrates the effectiveness of this method.Compared with the latest MMMA in 2022,the accuracy of the Test-Dev debugging test set in VQA v2 increased by 0.1%,and the Test-Std standard test set increased by 0.35%.(2)A method for dynamic knowledge fusion-based image semantic enhancement is proposed.In VQA,external common-sense can improve the model’s ability for implicit reasoning.To address the problem of performance degradation caused by a large amount of noise in the external knowledge introduced in existing methods,this method constructs strongly correlated knowledge triplets from open-source external knowledge graphs and uses weight adjustment to fuse external knowledge to enhance the semantic representation of image features,reducing the interference of redundant information from external knowledge on the model to some extent,and improving the model’s reasoning ability in answering questions.The effectiveness of this method is verified through testing on the VQA v2 and VQA-CP v2 datasets.Compared with the latest MMMA in 2022,the accuracy of the Test-Dev debugging test set in VQA v2 increased by 0.66%,and the Test-Std standard test set increased by 0.83%.(3)A life-scene-based VQA system is implemented,which integrates the joint interactive feature extraction and deep fusion method and the dynamic knowledge fusion-based image semantic enhancement method proposed in this thesis,demonstrating the practicality of the proposed methods in a real-world scenario. |
