Research On Knowledge-Enhanced Pre-trained Model Based On Graph Transformer

The Pre-trained Models(PMs)represented by BERT and GPT have made great progress in Natural Language Processing,Computer Vision and other fields.However,the PMs still face challenges including poor interpretability,vulnerability,large model size,weak reasoning ability,and commonsense errors.In the past two years,researchers have introduced knowledge into PMs to solve these problems.Most of the existing research on knowledge-enhanced PMs integrates knowledge in the form of entities or triples.This linear and flat structure lacks the deep multi-hop semantic information of knowledge graphs,resulting in incomplete knowledge integration.In addition,the semantic alignment and fusion of knowledge feature and text feature remains a challenge.To deal with the above problems,this paper proposes Graph Transformer-based Knowledge-Enhanced Pre-trained model(GT-KEPM),including a Graph Transformer-based knowledge graph embedding method with edge types and a knowledge and text feature alignment and fusion method based on contrastive learning.The work of the paper mainly includes the following three parts:(1)Propose a Graph Transformer-based knowledge graph embedding method with edge types.The method learns the semantic and directional information of entities and relations in the knowledge graph,and obtains a knowledge graph embedding representation containing multi-hop structural information.(2)Propose a Graph Transformer-based Knowledge-Enhanced Pretrained Model GT-KEPM.The Graph Transformer-based knowledge graph embedding method with edge types is used to obtain the knowledge graph node embeddings incorporating multi-hop structural information and the semantics of entities and relations.A contrastive learning-based knowledge and text feature alignment and fusion method is proposed to align knowledge and text features and integrate the knowledge graph node embeddings into the text embeddings to obtain knowledge-enhanced text embeddings,which improves the performance of the model on the generative commonsense reasoning task.(3)The feasibility and effectiveness of GT-KEPM are verified through experiments.GT-KEPM is compared with seven other models on six text generation metrics in the generative commonsense reasoning task.Compared with the pre-trained models without incorporating knowledge,GT-KEPM has achieved a 1.56%-21.55%improvement in six text generation metrics.Compared with K-BERT-Gen which incorporates triples,GT-KEPLM has achieved a 1.79%-10.15%improvement in six text generation metrics.Compared with the knowledge-enhanced pretrained model KG-BART,GT-KEPM has achieved 0.19%-7.35%improvement in BLEU-3,BLEU-4,ROUGE-2,ROUGE-L,METEOR,CIDEr,and SPICE metrics.Experimental results validate the performance of GT-KEPM.