Research On Entity Relationship Joint Extraction Method Based On Transformer Pre-Training Model

Entity relation joint extraction is an important basic task in existing natural language processing,which aims to identify entity and relation triples in sentences through template or deep learning.Existing researches mainly focus on pipeline and joint extraction,two deep learning research methods.Each of these two methods has its advantages and disadvantages,but the pipeline will have error accumulation.Therefore,joint extraction has been widely studied in recent years.However,the existing joint extraction methods often focus on the decoding layer and ignore the coding layer.And the current classic joint extraction method of decoding subject first and then decoding object and relation solves the problem of EPO and SPO,but there is error accumulation.,and unsatisfactory prediction results when some disturbance occurs to the data.So,the research is as follows:(1)In this paper,the classical joint extraction decoding model is used,that is,the subject entity in the triplet is first extracted,and then the object and relation are extracted according to the entity,and the object and relation are decoded simultaneously.In order to conduct a detailed study on the performance of existing Transformer pre-training model in relation extraction,this paper uses six pre-training models BERT,Bio BERT,Sci BERT,Ro BERTa,ALBERT and XLNet as encoders to conduct experiments on five published data sets.In order to ensure that the characteristics of different pre-trained models on relation extraction tasks can be verified,the data sets used in this paper include biomedical and scientific classes in addition to daily classes,so as to analyze the characteristics of the pre-trained models.XLNet and BERT show good coding ability,ALBERT and Ro BERTa have short training time.Sci BERT is not stable.(2)Aiming at the problem of error accumulation in relation and entity decoding of joint extraction decoding model,the training process and the testing process are inconsistent.We propose two decoding models,Subjects-object and joint annotation framework.The experimental results show that the F1 value of the relation extraction model based on the joint annotation framework is increased by 0.87%,2.11%,4.69%and 0.58%on ADE,Co NLL04,Sci ERC and Web NLG datasets,respectively.However,it is a pity that Subjects-object is not stable enough and only has good performance on a few data sets.The study in this paper achieves state-of-the-art performance on four datasets: ADE,Co NLL04,Web NLG,and NYT.(3)Aiming at the problem of weak generalization of the joint model,that is,when the data changes slightly,the accuracy of the judgment cannot be guaranteed.This paper proposes to use adversarial attacks to improve the robustness of the model.Four methods of adversarial perturbations,namely change_char,delete_char,exchange_char and add_char,are proposed to generate adversarial samples for adversarial attacks,and 10 different amounts of perturbations are used to do comparative experiments.The experimental results show that the model after adversarial attacks improves the Co NLL04,Sci ERC,Web NLG and NYT data sets by 2.41%,0.9%,0.72% and 0.23%respectively.It shows that the smaller the data set is,the better the effect of the adversarial attack is,and the larger the data set is,the less obvious the improvement will be or even reduce.