A Keyword-aware And BERT Pruning Approach For Document Relationship Extraction

Relation extraction is a core aspect of downstream tasks such as information retrieval,question and answer systems and knowledge graphs.Most of the current mainstream relationship extraction techniques focus on sentence-level corpus,which has significant limitations when deployed on the ground.In real-world applications,document-level corpora are more common and can contain richer information about entity relationships,which is of greater research value.However,the document-level corpus also poses a great challenge to existing relationship extraction models due to the large volume of data handled by the models,the large number of entities and the complexity of entity-relationship interactions.Currently proposed models based on graph neural networks or Transformer mostly focus on the entity representation of the global contextual information of the document,ignoring to a certain extent the impact of the really critical local information on the relationship extraction task.In reality,the relationships between pairs of entities can be inferred with only a few keywords,so the document-level entity relationship extraction task also needs to address the issues of how to filter the large amount of contextual input and extract the key information that can help classify relationships.Therefore,this paper conducts an in-depth study on document-level entity relationship extraction methods and pre-trained BERT model pruning methods,starting from refining contextual keyword features and streamlining document Token sequences as follows.(1)A keyword-aware contextual feature extraction method is proposed to address the problem that current mainstream relationship extraction methods encode the global context and key features are not prominent.The method is based on the Convolutional Long Short-Term Memory(Conv LSTM),which is applicable to text sequence processing,and introduces a Self-Attention Memory(SAM)module that can perceive key features and constitute the final the SAM-Conv LSTM is a keyword filtering model.The model uses subject and object entities to match the entire sequence of contexts,calculates an attention score for each input word against that entity pair,and perceives keyword features by finding word embeddings with high cross-attention by the entity pair.Finally the model uses the SAM module to update and record key local features to enhance downstream classification task performance.In addition,a modified loss function based on Circle Loss is used to address the positive and negative category imbalance problem in relational classification.Experimental results on three benchmark datasets(Doc RED,CDR and GDA)show that the model performs up to 0.87% better than current state-of-the-art methods in the F1 value for relationship extraction tasks in the general domain.(2)To address the problem of large data processing volume and long running time for document-level entity relationship extraction tasks,this paper starts from the input Token sequence of the pre-trained language model for model compression,and proposes a Token pruning method based on adaptive thresholding and a Token pruning method based on a lightweight classification module.The adaptive threshold-based approach uses a two-stage training process: firstly,a microsoftable mask is used to learn the importance score threshold and control the pruning rate of the model by introducing an additional regularisation loss,and then the threshold is fixed and the model is subjected to hard Token pruning and final parameter fine-tuning.The lightweight module-based approach uses an end-to-end training approach: each Token feature is first spliced with a global contextual feature,the spliced feature is fed into the prediction module to calculate the importance score that determines whether the Token stays or goes,and the module then ranks the scores and prunes the Tokens according to a set ratio.In this paper,the two proposed methods are fully tested on the GULE natural language task set.The experimental results show that the adaptive threshold-based approach and the lightweight module-based approach achieve up to 2.09 times and 1.44 times the amount of FLOPs operations compressed within 1% and 0.5% reduction in task accuracy,respectively.