Research On Relation Extraction Method Based On Pre-trained Language Model

Relation extraction is one of the key technologies for automatically building knowledge graphs,and its fundamental goal is to extract semantic relationships between entities.Relation extraction provides a technical foundation for subsequent tasks such as knowledge graph construction,intelligent information retrieval,and semantic analysis,and has a very wide range of applications.Early entity relation extraction methods relied mainly on the manual construction of relation extractors,which consumed a lot of resources.With the rise of deep learning,methods based on neural network models can effectively solve the problem of entity relation extraction,mainly because neural networks can automatically extract features,avoiding the cumbersome design of feature engineering.For unstructured and semi-structured texts,there are problems in the current relation extraction methods,such as inaccurate capture of contextual information and difficult utilization of sentence features.This thesis applies different Transformer hidden layers to represent the head entity and tail entity respectively,and integrates the local and global features of the text.We modify the downstream network architecture of the BERT pretraining language model,fully mining the advantages of multiple hidden layers of the pretraining model.We model the two entities(head entity and tail entity)separately in the relation extraction task,and use different methods for feature extraction.We design a novel relation extraction model and relationship calculation method.The main contributions and innovations of this thesis are as follows:(1)We propose a method that models the head entity and tail entity using different hidden layers.Based on the characteristics of the BERT pre-training language model,each hidden layer can output different hidden state vectors.In this work,we model the head entity and tail entity separately in the relation extraction task.In the entity embedding representation part of the head entity,we use sentence features as global features and capture contextual features in the hidden layers’ embedding representation using a multigranularity and equally-sized convolutional neural network.In the tail entity part,we use a deeper embedding representation that is homologous to the head entity’s.This work combines pre-training language models and relation extraction tasks more deeply and proposes a novel relation extraction model.(2)We design a novel relation calculation layer for the relation extraction model proposed in(1).Based on the homology of the head entity and tail entity embedding representation,we propose to calculate the similarity of the same position embedding representation in the two embedding matrices using asymmetric inner product to convert the relation extraction problem into a matrix classification problem,and design the corresponding loss function.Meanwhile,to solve the dimension inconsistency caused by word groups involved in the two entities in relation extraction,we design a novel masking matrix and solve the problem of multiple relationships involving the same entity in complex relations using multiple kernel matrices.(3)We conduct detailed experiments on the Sem Eval 2010 Task 8,NYT 10,and Wiki80 datasets based on the model proposed in(1)and design experiments targeting complex relation extraction tasks according to the dataset characteristics.We conduct comparative experiments with methods based on pre-training models and graph neural networks on the Sem Eval 2010 Task 8 and NYT 10 datasets,and with traditional convolutional neural networks on the Wiki 80 dataset,all of which demonstrate certain improvements.We conduct comparative experiments with mainstream models on the Sem Eval 2010 Task 8 and NYT 10 datasets,and conduct related experiments such as different hidden layer analyses,ablation experiments,and convolutional layer analyses,as well as complex relation analysis experiments designed based on the characteristics of the NYT 10 dataset,all of which verify the reliability of the proposed model on different types of datasets.