Transfer Learning Based Hybrid Neural Networks For Relation Extraction

With the increasing application scenarios of the Internet,massive informative text has been generated,which makes accurate knowledge acquisition more and more impor-tant.As an necessary step in information extraction,relation extraction task provides a powerful support for the construction of large-scale knowledge graph.Thanks to the rapid development of deep learning technology,the performance of relation extraction model has been significantly improved.The relation extraction models mainly adopt convolu-tional neural networks and recurrent neural networks as sentence encoders.However,it is difficult for convolutional neural networks to learn the long-range dependency between words,and the recurrent neural networks need more training time because they cannot execute in parallel.At the same time,most of the work takes the relation extraction as a single task,neglecting the relevance among related tasks,and wasting part of the semantic information.This paper studies and explores the above issues,and its contributions are summarized as follows:(1)In this paper,a novel hybrid neural network model is proposed,which combines piece-wise convolutional neural networks and entity-aware Transformer,so that the model can extract local features within sentences and learn long-range dependency features be-tween distant tokens.Entity-aware Transformer can leverage the semantic and syntactic information of a sentence and generate entity-specific representations.According to the fact that the sentence often contains a lot of irrelevant information for relation extraction,this paper uses the inner-sentence attention mechanism to grasp the relational tokens.Two popular datasets are adopted for experiments and the results conducted on NYT dataset show that the proposed hybrid model achieves an AUC score of 0.417,which is higher than baseline.(2)This paper uses a transfer learning strategy to utilize the prior knowledge learned from related tasks.Since the types of entities can effectively impose soft constraints of the relations between them,the results of entity typing task can be used in a reasonable way to improve the accuracy of the relationship extraction model further.In this paper,the entity typing task is chosen to train the entity-aware Transformer,and the model parameters obtained from the training stage are retained.In the relation extraction task,the entity-aware Transformer will be initialized with the prior knowledge learned from the entity typing task.The results on the dataset show that the transfer learning strategy can further improve the AUC score to 0.432.