Research On Automatic Text Summarization Based On Deep Learning

With the advent of the era of big data,the exponentially increasing amount of data makes people submerged in the sea of data.How to extract the information needed in the multitude of texts becomes extremely important.Automatic text summarization refers to extract the main information from the text,search for the key information,and condense the main information of the text into a concise summary in the way of using probability statistics,machine learning,deep learning,neural networks,etc.At present,it is widely used in the formation of news headlines,text retrieval,knowledge Q&A and so on.The sequence-to-sequence model is the most widely used model in natural language processing,which is generally composed of encoder and decoder.The text abstract is finally generated by representing and extracting features of the text in the encoder and by combining and expressing it in the decoder.However,due to the low quality of the training data set,inconsistent training and prediction,and the weak generalization capability of the model,there are some problems,such as Out-of-Vocabulary(OOV)words,repetitive text generation,poor readability,failure to express semantics and so on.In view of the above problems,this thesis studies the text representation,feature extraction,generation method and training process,proposes two text generation models,and solves the problems from multiple perspectives.It mainly includes the following points:(1)This thesis proposes a improved model that combines bidirectional long-term and short-term memory network with attention mechanism.Long-term and short-term mem-ory networks are used to extract text features and sequence location information,so the location coding information can be omitted.The unidirectional long-term and short-term memory network on the decoder side adopts initialization from the encoder side to the hidden layer state and cell state of the long-term and short-term memory network,which is beneficial to the fusion of context features.(2)This thesis studies the unlikelihood training process and reformulate training goals.The likelihood training process has the problem of self-reinforcement,and sometimes the problem of generating repetition occurs when the training and generating is inconsistent.Unlikelihood training puts forward the concept of negative candidate set and unlikelihood training target,which can reduce the problem of repetitive generation and the meaningless output.Decoding optimization on the generation side can make the quality of generated text more fluent and readable.(3)This thesis studies the characteristics of the pre-trained language model and ap-plies the Gibbs sampling algorithm to the masked language model.The pre-trained lan-guage model forms a relatively stable probability distribution,and the idea of Gibbs sam-pling process and the mask language model is highly similar.Therefore,the Gibbs sam-pling algorithm can be well combined with the masked language model to maintain uni-fication of the training and forecasts and improve the quality of text summarization.