Research On OOV And Long Distance Dependency Of Chinese Abstract Summarization Model

Automatic summary is an important tool for text parsing.It can extract the main information of text from massive data,and realize the rapid conversion from redundant text to concise text.Benifiting the rapid development of deep learning approach in Natural Language Processing(NLP),the abstractive summarization based on deep learning approach has also become a research hotspot in current abstractive summarization approach.For abstractive summarization,it is a sequence-to-sequence(seq2seq)mode,and the attention mechanism(Attention Mechanism)is fused to seq2 seq to achieve the focus on certain words in the input sequence Attention,we call that is seq2seq+attention framework.Therefore,based on the seq2seq+attention framework,this paper integrates the copy mechanism and Input-feeding method to improve the OOV(Out-of-Vocabulary)of the original model and the problem of attention determination accuracy.At the same time,the pre-training language model(Bert)and Transformer model are introduced to improve the problem of long distance dependence of sentences.The main research contents of this paper are as follows:(1)Consider that the seq2seq+attention framework needs to build a vocabulary first when generating the summarization.After the neural network learns autonomously,it selects the word with the highest probability from the vocabulary as the output.However,because the vocabulary is fixed,many words outside the vocabulary cannot be effectively generated,that is,the OOV problem occurs,so this paper proposed the copy mechanism to improve the problem.The vocabulary is dynamically changed through the copy mechanism to change the source.The words in the sequence are also taken into consideration,so that the output summary can include more words in the non-dictionary;then,because the attention of each moment in the original model is determined to be independent of each decoding step,we introduced the Input-feeding method connects the attention decision at each decoding step,making each word selected by decoder more accurate.(2)The RNNs(LSTM or GRU)model does not solve the problem of long-distance dependence in a true sense,so we introduced Bert+Transformer to improve this type of problem.This part is divided into Bert fine-tuning and Transformer to achieve two stages to generate a summary.In the first stage,learn the document-level features through Bert to obtain more semantic and grammatical information,and extractived sentences(Extractive),which can effectively shorten the length of the source text;in the second stage,input the sentence set in the first stage into the Transformer model,and Transformer model output the summary,which can effectively improve the long distance dependence of sentences.Using NLPCC2018 Chinese news text data for the experiment,and using ROUGE to evaluated,the experimental results show that the above method is compared with the classic extraction method and the results of the seq2seq+attention model,all ROUGE values have been improved,verifying the above method feasibility.In this paper did not use external knowledge to assist in the abstractive summarization,and not use Bert's improved models(such as ALBert)and so on,but in future work,we recommend these methods to improve the quality of Chinese summary.