Research Of Abstractive Summarization Via Graph-based Neural Network Model

With the advent of information technology,people have become more and more dependent on the Internet to obtain information in demand.However,with the development of the Internet,the information on it has exploded.How to effectively screen out the useful in-formation from the mass of information has become a key technical issue.Automatic text summarization technology can help users to retrieve relevant information from massive in-formation,avoid the problem of retrieving excessive redundant and one-sided information through search engines,and effectively solve information overload problem.At present,Automatic text summarization can be roughly divided into two categories:ex-tractive and generative.The basic method of extractive summarization is to extract some important sentences from the original text to form a summary.The research focuses on the important judgments,screening and ranking of sentences.The basic idea of generative sum-marization is to condense its ideas and concepts based on understanding the semantics of the original text to achieve semantic reconstruction.In recent years,with the rapid develop-ment of big data and artificial intelligence technology,traditional automatic summarization research is evolving from extractive summarization to generative automatic summarization to achieve the purpose of generating higher quality summarization.Generative summarization can be classified into structure-based and semantic-based meth-ods.The main disadvantage of the former is that the language quality is relatively poor,for example,the sentence contains more grammatical errors;The summary generated by the lat-ter is concise,cohesive,informative and low redundancy,but the main disadvantage is that it mainly uses thousand-layer natural language processing technology.In recent years,deep learning technology has provided new ideas for automatic summarization research.Graph neural network models have achieved good results.In addition to being longer than a single sentence,the structure of a single document sum-mary is more complex.Moreover,its structure is often closely related to the importance of the information in the text.How to effectively use the text structure information to identify and represent the important information of the text is a major challenge faced by the text summarization.In addition,another challenge of text summarization is how to generate the summarization accurately and concisely based on the important information representation of the text.In this thesis,the two major challenges for the targeted research,specifically,this thesis has three main contributions.1)The graph-based text representation method is applied to the sequence encoder,which can make full use of the structural information of the original text and realize the recognition and representation of the text information.The existing models generally only use a single-layer codec framework for text summarization.This works well for short text inputing(such as sentence reduction tasks),but for long text input(such as the more common chapter in a text summary),the processing ability becomes very limited.Generally speaking,for long text input,the solution of the single-layer model is more through truncation,that is,only a certain segment of the input text is taken for summary generation.This approach has two major disadvantages:one is that it is highly dependent on the way of truncation,and the other is that it will bring certain errors for the recognition of important information in the text.In contrast to single-layer encoders,hierarchical encoders consider the input text as a sequence of sentences rather than as a sequence of words.Accordingly,the coding process is divided into two steps:firstly,each sentence in the text is encoded based on the word,and then the whole text is encoded based on the sentence.However,it is only layered on the encoder,but not on the decoder.Another of the more important question is,for text level,in addition to the influence of the length,it also contain the very complicated structure,and these models are based on sequence modeling was carried out on the text structure,this is not good for important information extraction in the input text,which was the text representation model based on graph compared with the advantages of these sequence model.In the text representation based on the graph,firstly,the sequence is taken as the input in the sentence encoder,and the output result is an initial sentence representation.Then the text is regarded as a sentence sequence input by the document encoder,and the output result is the final document representation.The sentence encoder and the document encoder consti-tute a hierarchical encoder.The sentence encoders and document encoders in this thesis are implemented by using single-layer one-way recurrent neural network.Finally,there is the graph-based sentence representation module.The module can update the representation of each sentence based on the input text structure information.In the graph,each node cor-responds to a sentence in the original text,and the edges between nodes correspond to the relations between sentences.The constructed graph is described by the adjacency matrix.By adding self-join to the adjacency matrix,the original information of each sentence can be retained after updating.In this thesis,a single layer convolutional neural network is used to update the sentence representation.In the analysis of the effect of different text repre-sentation on abstract performance,graph-based representation can effectively improve the performance of the summarization.2)This thesis introduces a salient sentence extractor into the generative summarization model.Because the graph-based representation may not be able to effectively identify important in-formation without the guidance of supervision information.In this thesis,a salient sentence extraction module is introduced.This module can filter out some unimportant information in the input text,select the salient information in the original document more effectively,and improve the accuracy of the summarization.The module consists of a feedforward neural network.The input consists of three parts.The first part is the sentence representation after the text representation model is updated based on the graph.The second is the document vector encoded by the document encoder,and the third is the position embedding of each sentence.In addition,the extraction supervision information is introduced to the salient sen-tence extraction module.That is,each sentence in the input text is assigned a specific label to indicate whether the sentence is an salient sentence.This supervision information can us6 the extraction label to carry on the supervision training to the extraction sentence module.In this experiment,the CNN/Daily Mail data set was adopted.During the ablation exper-iment,it was found that the performance of the model would decrease to a certain extent after the graph-based text representation module or salient sentence extraction module were removed.When comparing the efects of different generative supervision information on summarization,the joint model proposed in this thesis is compared with the label extraction-methods commonly used in the generation summarization.This method is to retrieve the sentence sequence from the original text which can be maximized by the ROUGR index at the text level according to the existing generative summarization.The comparison results show that the model ROUGE score in this thesis is higher,that is,the extractive label can get better performance.3)In this thesis,the generation process of generative summarization is improved and the model is optimized by combining the advantages of extractive summarization with the pro-cess of generative summarization.Extractive summarization occupies a dominant position in traditional text summarization because of its simple structure and its sentences are from the original text,which can guarantee the accuracy of information to some extent.In this thesis,a joint summary model is proposed to improve the generative abstract model.Extrac-tive and generative summarization are considered as two different tasks in this thesis.The model is divided into two layers:sentence decoder and word decoder.Among them,the scope of attention mechanism used in sentence decoder is based on the sentences extracted from the salient sentence extraction model.In the comparison with other generative summarization models,the overall effect of ROUGE scoring was higher.In terms of ROUGE-2 index,the performance of pointer model was slightly better than that of this model.The possible reason is that the hierarchical model did not directly generate summarization at the word level,which affected the summary perfor-mance to some extent.