Research Of Open Relation Extraction Based On BERT Pruning Model

Relation extraction is an important branch of natural language processing with practical significance in tasks such as web search,knowledge base construction and question answering.Traditional relation extraction refers to identifying a pair of entity concepts and connections from unstructured text,where its relation type is limited,and when targeting new domains,the data needs to be re-annotated,which is time-consuming and tedious.In recent years,researchers have worked to explore open relation extraction to address this problem.The paper investigates the problem of open relation extraction using pre-trained models.The main research content and innovation points are as follows.(1)An open relationship extraction model based on optimized entity pseudo labels is proposed to address the problem of low quality of relationship pseudo labels generated by clustering in most current open relationship extraction models.The model first performs supervised pre-training on the predefined relations in the training set and their existing labeled instances to achieve a non-linear mapping optimization,through which the high-dimensional entity pair representation is converted into a relation-oriented representation to improve the quality of the pseudo-labels.Secondly,an unsupervised comparative learning method is designed to enhance the unlabelled data,resulting in better clustering of entities of the same class.Finally,a model is jointly trained using supervised and unsupervised data to iterate over entity pair representations.The proposed method is evaluated on two publicly available open relational extraction datasets,Few Rel(Few-Shot Relation Classification Dataset)and TACRED(TAC Relation Extraction Dataset).The experimental results show that the method scores better than most of the current state-of-the-art methods on the three clustering performance metrics.(2)To address the problem that BERT(Bidirectional Encoder Representations from Transformers)pre-trained models are difficult to deploy to computationally constrained devices,an open relational extraction model based on iterative weight pruning of BERT is proposed.First,the multi-head attention mechanism of the BERT model is explored,and the weight focus of different attention layers and different attention heads and their contribution to the overall model performance are analyzed.The importance of different attention layers for grammar analysis is also experimentally evaluated and analyzed.Secondly,a new iterative pruning method is proposed,which uses attention weight pruning and model knowledge distillation to generate BERT pre-trained language models with high sparsity.Random pruning,hierarchical pruning and iterative pruning experiments are conducted on the open relational extraction datasets Few Rel and TACRED respectively.The experimental results show that the models obtained by pruning are able to achieve high sparsity while minimizing accuracy loss.(3)Design and implement an open relational extraction system platform around the relational extraction model in this paper.The platform is based on Py Torch and currently popular front-and back-end development techniques to achieve dataset management,relation extraction and relation visualization.Firstly,the system can display the datasets currently available in the platform,their data and size,relationship types and other specific information,and users can upload new datasets to extend them according to their requirements.Secondly,the user can select the corresponding dataset and open relationship extraction model for relationship extraction demonstration through the relationship extraction function.Finally,the entity-relationship triad information generated by the relationship extraction is visualised by building a knowledge graph.