A Transferable Approach To Generating Abstractive Text Summary Based On Pre-trained Language Model

In recent years,with the rapid development of deep learning,natural language processing has become one of the key areas of scientific research.At present,the research work on processing natural language based on deep learning technology is mainly concentrated in the areas of knowledge graph,machine translation,question answering system,text classification,etc.However,there is very little research work on applying deep learning technology to automatically generate text summary.In addition,in today's so-called ”Internet era”,the daily production of text information has shown an exponentially explosive growth trend.Faced with text information of magnitude beyond imagination,humans are no longer capable of extracting the core ideas they want to convey from the text.Therefore,in order to be able to efficiently extract the core content from the text information,that is,text summary,we urgently need an automatic text summary generation technology based on deep learning,which is the focus of this paper.Text summary is one of the downstream tasks of natural language processing,and is one of many generative tasks.It is divided into extractive text summary and abstractive text summary.Compared with extractive text summary,abstractive text summary is more readable and fluent.In addition,in related research work,it is found that many of the current downstream tasks of natural language processing are mostly implemented by fine-tuning based on pre-trained language models.Excellent pre-trained language models can fully understand the context and generate word representation that contains the context.This plays a very important role in improving the performance of downstream tasks.Therefore,this paper aims to implement an abstractive text summary generation model based on pre-trained language models,namely the REINS model.The model will adopt the mainstream Encoder-Decoder architecture idea for generative tasks.The Encoder will be built based on the currently most popular pre-trained language model BERT,so the REINS model can make full use of context information in word representation to help the Decoder better decode;the Decoder will be based on the Decoder component in the machine translation model Transformer.This component abandons the ideas of RNN or CNN commonly used in traditional generative tasks,and uses the attention mechanism to help the REINS model to pay more attention to the internal relations at the word level and sentence level,making the model perform better in abstractive text summary tasks.In addition,compared with the traditional text summary generation model based on RNN or CNN,the REINS model also supports parallel mode,which can speed up the training,evaluation,and inference process of the model.Finally,like the model for other downstream tasks based on BERT,this model will have better transfer capability,that is,it can obtain better text summary generation quality by training with fewer data sets.The main work and contributions of this paper include: first,this paper proposes a transferable abstractive text summary generation model;second,this paper builds an abstractive text summary generation system based on this model,namely the TATSGS system,which can provide a series of pipeline services such as data preprocessing,model training,model evaluation,and model application;Finally,this paper uses the standard text data set CNN&Daily Mail to verify the performance of the REINS model in generating abstractive text summary,and obtains good experimental results.ROUGE-1,ROUGE-2 and ROUGE-L respectively reached 40.13,17.87,36.94,thus verifying the transferability of the text summary generation model REINS built based on the pre-trained language model BERT.