Research On Transfer Learning Based Relation Extraction Methods

Relation extraction aims to extract entity relation information from free text.The en-tity relation information contains rich commonsense knowledge of reality,which is help-ful for machines to understand the deep semantic information of text,so as to complete a series of natural language processing tasks that are closely related to reality.As a key part of the automatic extraction,representation,and application of structured knowledge,re-lation extraction is not only an important subtask of information extraction,but also a key technology for the construction of knowledge graphs.Therefore,it has attracted extensive attention and has been developed rapidly.Especially in recent years,the application of deep learning methods such as pre-trained language models has greatly improved the per-formance of various relation extraction models.Although the current deep learning-based relation extraction methods have achieved impressive results,there are still many serious challenges.Starting from perspectives of extraction methods,application scenarios and extraction objects,these challenges can be summarized as follows:1)In terms of extraction methods,the existing relation extraction models are mainly based on the pre-trained language models,while the traditional pre-training language models are difficult to solve knowledge-driven tasks.So even the subsequent improve-ment works add complex structures to them,they are still difficult to achieve further per-formance breakthroughs.Therefore,a method that can effectively improve the perfor-mance of pre-trained relation extraction models is needed;2)In terms of application scenarios,the deep learning framework requires large-scale labeled samples to train the model.When there are only a small number of samples,the accuracy of the existing relation extraction model will be severely declined,so a method that can solve the low-resource relation extraction task is needed;3)In terms of extraction objects,the mainstream relation extraction works only focus on the relation in a single sentence.When facing more complex document-level relation extraction tasks,the existing relation extraction models often perform poorly.So a method that can effectively deal with document-level relationship extraction is needed.The above issues seriously hinder the development progress and practical applica-tion of relation extraction technology.The fundamental reason is that the actual situation faced by the relation extraction model is quite different from the prerequisites required by the deep learning framework.That is,it is difficult to obtain both quantitative(large-scale)and qualitative(same-distribution)data in reality.So it is also difficult to meet the requirements of deep learning for large-scale labeled data and the same distribution of samples,resulting in a serious decline in model performance.Transfer learning can use the similarity between data,tasks or models to transfer the knowledge learned in specific fields to other fields,so as to loosen the dependence on large-scale labeled training data and the same distribution hypothesis to a certain extent.However,the implementation process of transfer learning is far from trivial.It is necessary to select appropriate transfer strategies according to the transfer objects and design a reasonable model framework ac-cording to the characteristics of tasks.Therefore,this thesis focuses on the three basic el-ements of data,model,and task in the deep learning framework,and selects three transfer learning methods,namely,knowledge enhanced pre-trained methods focusing on trans-fer learning between different model parameters,few-shot metric meta-learning methods focusing on transfer learning between different tasks,and multi-stage domain adaptation methods focusing on transfer learning between different sample distributions.Then com-bined with the application scenarios of various methods and the characteristics of different tasks,three model frameworks are designed to complete the relation extraction task based on the transfer learning methods,namely,Entity Knowledge Aware pre-trained language model(EKA),Interactive Attention Network model(IAN),and Multi-stage Adaptive Fea-ture Alignment model MAFA.Specifically,the main contents and contributions of this thesis can be summarized as follows:1)Aiming at the issue that the pre-training language model-based relation extraction models have a complex structure but limited performance,based on knowledge enhanced pre-trained methods which focus on transfer learning between different model parameters,an EKA model for applying knowledge enhanced pre-training to sentence-level relation extraction task is designed.Since the knowledge-enhanced pre-trained language model is a new direction that is still in the exploratory stage,and it has not formed a strict definition and a unified framework.So this thesis first comprehensively summarizes its background,basic concepts,and mainstream methods,and analyzes the key elements affecting its per-formance,so as to make up for the blank of comprehensive review articles in this direc-tion.Subsequently,this thesis designs an EKA framework that introduces entity-related knowledge from three layers: the input features,the decoders,and the training tasks for pre-training language models.And it uses sentence-level relation extraction data sets TACRED,TACREV,and Re-TACRED to evaluate this method.A large number of ex-periments have proved that the knowledge-enhanced pre-trained model can significantly improve the accuracy of relation classification without adding complex structures.2)Aiming at the serious degradation of model performance under low-resource con-ditions,based on few-shot metric meta-learning methods which focus on transfer learning between different tasks is proposed,an IAN framework using an interactive attention net-work to complete few-shot relation extraction tasks is designed.Under this framework,the model can learn meta-knowledge that can be transferred across tasks by completing different types of meta-learning tasks.This meta-knowledge can guide the model to cap-ture the inter-instance and intra-instance interactive information,and update the instance representation based on the interactive information to make it more representative.A large number of experimental results on the FewRel,a few-shot relation classification dataset,prove that this method can achieve high classification progress on the condition that there are only a few labeled samples are available for reference.3)Aiming at the issue that the model is difficult to deal with the complex document-level relation extraction task,based on multi-stage domain adaptation methods which focus on transfer learning between different sample distributions is proposed,a MAFA framework which can use the multi-stage feature alignment method to complete the document-level relation extraction task is designed.The framework implements the filtering of noise data and the feature alignment of different domain data in two stages,so as to fully train the model by using multi-domain data.In the first stage,a denoising labeler is trained by a manually annotated dataset,and the labeler is used to complete the noise reduction and screening work of a distantly supervised dataset.In the second stage,the model is fully trained with multi-domain data,and the adaptive module is introduced to realize the domain feature alignment,so as to adaptively adjust the tendency of the model to dif-ferent domains during training.A large number of experimental results on DocRED,a document-level relation extraction dataset,prove that this method can effectively improve the scores in the complex document-level relation extraction tasks.In summary,this thesis has carried out a series of researches based on transfer learn-ing technology; focusing on relation extraction tasks; using knowledge-enhanced pre-training,few-shot metric meta-learning,and multi-stage domain adaptation methods; ori-ented to sentence-level relation extraction,few-shot relation extraction,and document-level relation extraction tasks.In the research process,we overcame a variety of diffi-culties such as how to further improve the scores of pre-trained language model-based relation extraction methods,how to complete the relation extraction task under low re-source conditions,and how to use data from different domains to jointly train the model for solving the complex document-level relation extraction tasks.And this thesis has also made remarkable achievements in multiple related datasets,such as TACRED,TACREV,Re-TACRED,FewRel,and DocRED.In short,this thesis focuses on how to use transfer learning to further improve the score of relation extraction tasks,especially relation extraction tasks under complex con-ditions such as insufficient annotation data or inconsistent sample distribution.