Research On Chinese Text Summary Generation Based On Pre-trained Language Model

With the advent of the information age,more people rely on the internet to get the information they need,such as user reviews,news reports,blogs and other types of social media,there is a huge amount of redundant data in this textual material,and it becomes especially critical to process these textual resources.If it is all human to analyze each text to generate a summary,it will take a lot of resources,which is even an impossible task.Automatic text summarisation techniques in the field of natural language processing provide an idea to solve this problem.Pre-trained language models are trained on a large scale corpus to learn general patterns of language and then trained on a specific downstream task-related corpus to match the requirements of the downstream task.In this paper,we study the characteristics of each pre-trained language model and try to apply them to the task of Chinese text summary generation.(1)Constructing a baseline Chinese text summarization model based on the Seq2 Seq structure and combining pointer generation network and reinforcement learning.The current Chinese text summarization model based on Seq2 Seq structure still suffers from the problem of exposure bias,resulting in its poor performance of text summarization model.In this paper,we design and implement a Chinese text summarisation model based on the Seq2 Seq structure and combining pointer generation network and reinforcement learning(PGN+RL).The loss function of the traditional language model is weighted with the loss function of reinforcement learning to reduce the problems caused by exposure bias.The experiments show that the PGN+RL text summarisation model constructed in this paper can effectively improve the quality of model generation,and it is used as the baseline model for this paper.(2)Construction of Chinese text summaries based on each pre-trained language model.By investigating several pre-trained language models,this paper attempts to apply each pre-trained language model to the Chinese text summary generation task.Experiments were conducted on the LCSTS dataset and summary examples were generated for comparative analysis.It was found that the GPT-2 and Uni LM pre-trained language models performed better compared to the baseline model,while the BERT and ERNIE-GEN pre-trained language models performed worse in generating summaries.(3)Construction of an improved Chinese text summarisation model based on GPT-2.The performance of the GPT-2 pre-trained language model is improved on the Chinese text summary generation task by improving the GPT-2 pre-trained language model in the encoding stage and data pre-processing stage.Comparing with the baseline model and other pre-trained language models on the dataset LCSTS,the experiments show that the improved model in this paper achieves the best ROUGE score.By comparing the model-generated summary examples,it is shown that the improved model in this paper performs better in generating summaries in terms of readability,coherence and inclusion of sentence completeness,verifying the effectiveness of the model-specific improvements.Comparing the performance of the models on the new text summarisation dataset NLPCC further validates that the models proposed in this paper have some generalisation capability.In summary,this paper demonstrates that the improvements made to the GPT-2 pre-trained language model can be better applied to the text summarisation task and can further improve the quality of the summaries generated by the model.