Modular Controllable Text Generation Optimization Method Based On Pre-Training Model

Text generation is an important task in natural language processing.In recent years,language models have occupied the dominant position of natural language generation.Moreover,as social media has produced a large amount of natural language data,the pre-trained model,which is widely used in several downstream tasks,have received great interests.However,it becomes an urgent request to control the generated text of pre-trained model with specific attributes.At present,there are three types of methods in the field of controllable text generation.(1)Guided Encoding based approaches can generate text through fusion and replacement during decoding,but the semantic meaning is stiff and grammar errors may occur.(2)Domain-Specific text generation methods benefit from training a large-scale pre-trained model by specific data,but the training cost is relatively high and the universality is not strong.(3)Fine-tuning language models can generate natural language text with specific attributes,but it will require a deep and finer-grained parameter tuning which makes these models less practical.Based on the above problems,in this paper,we proposes a fast and controllable text generation framework.The main contributions are as follows:(1)We propose a fast and controllable test generation framework.Based on the large-scale pre-trained language model,we propose a combination model which includes the gradient return method and the residual module to control the attribute and efficiency of text generation respectively.Since the pre-trained model and attribute model can be freely matched,the combined framework has the ability of fast,controllable and generalizable text generation.(2)We conduct empirical evaluation on two subject of text generation: emotion and theme.We separately apply the emotion and theme control module on long-and shorttext pre-trained model to evaluate the effectiveness of the framework.Experimental results show that the combined framework is capable of ”plug-and-play” of different pre-trained models and attribute models,and the average accuracy of target attribute text generation can reach 81% and the average time of text generation can be reduced by 78%.