Design And Implementation Of Visual Question Answering System Based On Knowledge Graph

The existing visual question answering technologies can only answer more accurately questions that can be answered directly from the image.For answering more complex questions,current visual question answering systems lack relevant data sets,and their answers are uninterpretable.This paper designs a visual question answering system based on the knowledge graph,which can answer more types of questions.The user asks a random question for a given image,and the system can give the corresponding answer.By differentiating the types of questions,the system enables different types of questions to be answered in the most appropriate way.The questions in visual question answering can be roughly divided into two categories,one is the question that can be answered directly through the visual information of the image,and the other is the question that needs to be answered with external knowledge.For the second type of question,the triple knowledge extracted from the entity description text and related knowledge graph is used to answer the second type of questions.The main research of this system includes the following three aspects:1)We propose the answering method of the question template that needs to be answered with external knowledge.We use the classifier to classify all the problems and filter out such problems.The answer is based on a template,and different templates correspond to different SPARQL query templates.With the help of the image entity information and related information in the question text,the answer to the question can be searched in the data source.The question template is divided into two categories according to whether it contains a clear entity to be queried,and different entities and attribute acquisition methods and answer query methods are designed to answer the question more accurately.2)We design a matching template method for questions that need to be answered with external knowledge.The question in Visual question answering use natural language to ask questions.There are many types of questions.In most cases,the question contains more than one entity or one attribute.The question template is the simplest question style,containing only one entity and attribute.In order to match the problem template accurately,we design an algorithm of the problem matching template,and complete the matching of the problem template.3)According to the relevant knowledge of the knowledge graph,we design a method to query answers from data sources.The key to the questions that need to be answered with external knowledge is what external knowledge is included and how to obtain answers from external knowledge.An external data source is created.The data source is composed of triple knowledge extracted from the entity description text and related knowledge graphs.SPARQL query statements are used to query the data source.We design a visual question answering system based on the knowledge graph in the animal field,which can answer questions raised based on the images.