Research And Implementation Of Intelligent Algorithms For Open-ended Text Generation

With the enhancement of language model modeling capabilities,it is no longer a problem to use machines to generate fluent sentences or fragments.However,when the length increases,the generated text is difficult to maintain the original high quality,and it is beginning to appear serious inconsistencies and degeneration problems.The fundamental reason is that the language model has deviations in the text modeling,and there is no assurance that the predicted probability distribution will always conform to the context,resulting in the generated words inconsistently or irrelevant to the previous text.The second is that the decoding algorithm does not reasonably avoid the deviation of the language model,so inconsistent or irrelevant problems gradually accumulate as the length increases,and deviates from the original quality.Based on the above reasons,this thesis has researched on improving text consistency and the quality of long text generation from two aspects: language model and decoding algorithm.The main contents are as follows:(1)On the basis of Transformer-XL,improved and designed a controllable long text generation model based on global memory,it divides the memory of Transformer-XL into two parts: local memory and global memory.Which inherits the long text modeling advantages of Transformer-XL,and also realizes controllable text generation.The dependence of content text on the same control information has also improved consistency and the quality of long text generation.(2)Proposes the decoding strategy of Sampling-based Heuristic Tree Search,which includes two components: a generator and an evaluator.The generator is used to generate sentences and the evaluator is used to score sentences.The algorithm controls the generation in sentence units,and sentences with low scores will be discarded in subsequent decoding.Benefited from the additional evaluator scoring and a unique fallback mechanism,the Sampling-based Heuristic Tree Search algorithm can avoid the accumulation of bias in the generated text as much as possible,which has great advantages when generating long texts.(3)Proposes algorithms of Nucleus Sampling with Temperature and Diverse Beam Search with Sampling for improving consistency and flexibility,respectively.Where the Nucleus Sampling with Temperature is an improved version of the Nucleus Sampling algorithm,which greatly improves the consistency at the cost of adding only a line of code;the Diverse Beam Search with Sampling solves the problem that Diverse Beam Search always produce fixed output,and improves the flexibility of generation.In addition,this thesis also designs an improved text repetition penalty strategy,which dynamically calculates the penalty factor according to the degree of repetition,so can more thoroughly avoid the problem of text repetition.(4)A comprehensive experimental evaluation of the above methods has been carried out,which proves the effectiveness of the improved method.When the improved model and the improved algorithm are applied at the same time,the BLEU score is increased by about 33% compared with GPT2,and the gap of the consistency score between the human text is reduced by an order of magnitude compared with the baseline.Finally,based on the above research,an open-ended text generation system is designed and implemented,which integrates multiple decoding algorithms and can free to choose when generating.The system also provides model training and text evaluation functions,it can meet the basic needs of various open-ended text generation tasks.