Research On Abstractive Text Summarization Technology Based On Pre Training Model

Automatic Text Summarization(ATS)aims to accurately select the key information from text,condense the core content,assist readers in improving their screening efficiency,and generate fluent and readable summaries in an automated way.With the development of largescale corpora and deep neural networks,ATS has made significant progress.However,there is still a considerable gap between the quality of automatic and manual text summarization due to the weak ability of current models to capture cross-sentence structures and multi-level semantic information.Thesis proposes a generative summary model based on pre-trained models,which deeply mines entity information,topic information,and structural information from the text,constructs a heterogeneous graph neural network to enhance the model’s ability to select information,introduces contrastive learning to improve the problem of pre-training model exposure bias,and effectively improves the quality of automatic text summarization.Thesis focuses on the field of single-document summarization.Compared with traditional text summarization techniques,the main contributions of Thesis are as follows:(1)A multi-information-degree heterogeneous graph model.Based on the traditional graph neural network text summarization technology that only models sentence nodes,the model further models the dimensions of nodes from the sentence level to the word level and local and extended information,and establishes sentence-entity-sentence(ETS)and sentence-topicsentence(TTS)edge structures,constructing a text graph network centered on sentence nodes to extract deep,cross-sentence,and implicit associations in the text.(2)A double-layer attention mechanism heterogeneous graph network model for extractive summarization tasks.Based on the multi-dimensional node modeling information,the model constructs a multi-level attention mechanism,including node-level attention and semantic-level attention.The node-level attention mechanism learns the importance of adjacent nodes of sentence nodes under the same edge structure,and the semantic-level attention mechanism can learn the importance of different edge structures.Meanwhile,Thesis integrates global sentence features into the node-level attention mechanism to enhance the model’s learning ability.(3)A scoring model based on contrastive learning.By constructing a reference-free summarization scoring model that approximates the evaluation metrics that require reference summaries,the model learns the comparative sorting scoring model of evaluation metrics directly.This solves the problem of inconsistent training objective functions and evaluation metrics in pre-training language text summarization models and helps to improve the quality of summarization.