Research On Evaluation And Verification Of Multimodal Knowledge

Traditional knowledge graph mainly uses triples which are extracted from tables or text data.With the development of related technologies,research on constructing multimodal knowledge graph with images and data in other modal has emerged.In this process,noise and conflicts are inevitably introduced into knowledge graph.In order to apply the knowledge graph more effectively,it is necessary to evaluate and verify the multimodal knowledge.Different methods are used to evaluate triples from different modals.The knowledge obtained from text is evaluated based on background information and the context information of the triples,and the knowledge obtained from image data is evaluated based on scene graph.The following research has been done in this paper:1)This paper designs a knowledge evaluation method based on background information which is named PTCA.Knowledge representation learning combined with background information(the strength of association between entities,entity type information and multi-step path information)is utilized to evaluate triple knowledge from text.The confidence calculation ability of PTCA is evaluated by triple classification task,knowledge graph noise detection task and knowledge graph completion task.The experimental results show that PTCA can screen out triple knowledge with high reliability and perform better on high noise data.2)This paper designs a knowledge evaluation method which is named TCA based on the context information of triple.This method evaluates triples from text utilizing knowledge representation learning combined with triple context information(internal path information of knowledge graph and external text information related to entities in triples).The confidence calculation ability of TCA is evaluated on theknowledge graph noise detection task,knowledge graph completion task(entity link prediction task)and triple classification task.The experimental results show that TCA has good noise processing ability and knowledge confidence calculation ability,and has a stronger ability to evaluate low noise data.3)A knowledge evaluation method which is named NMEG based on scene graph is designed in this paper,which uses image information,image context information and external knowledge graph information to evaluate the confidence of triple generated from images.The confidence calculation ability of NMEG is evaluated in predicate classification task,scene graph classification task and scene graphgeneration task.The experimental results show that NMEG can effectively calculate the confidence of knowledge,and obtain higher quality knowledge.