Research And Application On Controllable Text Generation Based On Pre-trained Language Models

With the enhancement of the modeling ability of the pre-trained language model,it is no longer so difficult to use machines to generate smooth and reasonable sentences.However,when the length of generated text increases,the content and style of the text are no longer easy to control and prone to degradation.Based on the existing theories and technologies,we expand and study the content control and style control of controllable text generation.The main work is as follows:（1）Aiming at the problem of content control,we use keyword extraction algorithm to construct content prompt data set after crawling special corpus.Based on the existing pre-trained language models,we propose dynamic loss optimization to improve the unidirectional autoregressive language model and an encoder independent enhancement method to improve the encoder-decoder model.The experimental results show that the optimization method in this thesis improves the BLEU score by 2.27% and 5.72%respectively.（2）Aiming at the problem that the style is difficult to control,we propose an implicit variable decision-making method by introducing the generative adversarial network into the encoder-decoder model architecture and optimizing the training process.We construct the data set of text style control through syntactic analysis,and design an auxiliary style implication dichotomy experiment.By comparing the existing model algorithms,the experimental results show that the optimization in this thesis improves the probability through the classification model by 7.11%.When the generated text length increases to 100,the PPL can be reduced by 6.60% compared with that before improvement.（3）Aiming at why the classification model makes the decision,we introduce a model interpretation algorithm LIME to the text style classification model.Through the integration and improvement,the interpretability granularity is increased from the traditional word level to the sentence level,so as to build a more targeted data set.We also propose the “whole sentence masking” strategy to improve the pre-training task to enhance the understanding ability of the classification model to sentence semantics.Comparative experiments show that the optimization method in this thesis can improve the F1 score of classification task by 2.67% and the BERTScore of generated text by12.66%.（4）We analyze,design,and implement a controllable text generation system.The system is based on B/S architecture,the front end is based on browser page,and the back end is developed based on python and Flask web framework.Users use the text generation model trained in this thesis to assist text writing through friendly interactive pages,which has high usability.The administrator can manage the corpus and model in the background management interface to optimize the performance of the model in the system.In this thesis,we show the algorithm process and the key parts of the system implementation,and verify the effectiveness of the algorithm and the availability of the system through experiments.