Research On Construction Of Knowledge-infused Pre-trained Language Model For Abstract Of Scientific Papers

Abstract of scientific papers is a general statement of the key contents of scientific papers,which carries rich scientific knowledge and has the characteristics of academic language expression.At present,many researches on scientific intelligence are based on the abstract text of scientific papers.How to improve the mining ability of abstract text of scientific papers is an important problem in information science research.As a key component of natural language processing research,pre-trained language model can greatly improve the basic ability of text mining.This study attempts to construct a knowledge-infused pre-trained model of the abstract text of scientific papers,which can effectively support the mining and application of abstract text of scientific papers.Based on the analysis of pre-training model construction and other related research,aimming at the practical needs of classification problems(important sentence recognition,topic classification)and sequence annotation problems(entity recognition,keyword recognition)involved in text mining of scientific paper abstracts,and considering the reality that scientific information institutions have a large number of knowledge resources(such as glossary and thesaurus),this study puts forward a scheme to build the pre-training model,which is to complete the further pre-training of BERT model by introducing a large number of abstract texts of scientific papers in the way of "Continual Pretraining".The research focuses on two aspects:(1)Improving the pre training objectives to realize the adequate learning of the knowledge features of the abstract text;(2)Expanding the structure of the pre-trained model to effectively inject the content of syntactic-semantic knowledge.Under the above scheme,this study focuses on three key issues:(1)Designing of training objectives in the process of continual pre-training;(2)Injection representation of external knowledge for model structure augmentation;(3)The injection computation of knowledge vector in model structure augmentation.In the light of the above three key issues,three innovative attempts are carried out:(1)Two new pre training objectives(Term Masked Language Model and Move Sentence Order Predication)are proposed to realize the adequate learning of the knowledge features of the abstract.(2)A joint embedding and encoding knowledge vector representation method is proposed to realize the embedding representation of tags-like external knowledge vector and the encoding representation of triples-like external knowledge vector.(3)A model structure integrating knowledge vector injecting calculation is designed to realize the injecting calculation of tags-like knowledge vector and triples-like knowledge vector.This dissertaion conducts pre-training experiments based on the copus of the abstracts of scientific papers in Chinese medical domain,and validates the effectiveness of the pre-trained model on several datasets of classification task and sequence labeling task in Chinese medical domain by fine-tuning.The experimental results show that the model constructed in this paper outperform the BERT-model by 1.4%,2.75%,4.96% and 5.1% in four Chinese medical scientific papers mining task(move recognition,document classification,keyword recognition and entity recognition),also outperforming other models in related research such as Mac BERT,Med BERT and MCBERT.The above research shows that the pre-trained language model of scientific papers constructed in this study can effectively improve the mining effect of classification and sequence annotation tasks in scientific paper abstract text mining,and has application value for the subsequent related mining and analysis of scientific paper abstract text.