Neural Network Approaches For Style-constrained Controllable Text Generation

Text generation plays a crucial role in various modern applications of natural language processing,with the objective of crafting smooth and meaningful natural language text.Style-constrained text generation focuses on modeling and manipulating the style of generated text,representing a category of controllable text generation.Broadly speaking,”style” refers to any feature of the text that needs to be controlled,which could encompass tone,grammar,and content attributes.By fine-tuning these attributes,models can generate stylized text adapted to specific scenarios.However,the end-to-end generation mechanics of neural network models,coupled with the complexity and diversity of language itself,makes the precise control of the content,intent,and style of generated text exceedingly challenging.This requires the model to fully grasp the fundamental rules of human society from limited and heterogeneous data-a feat even large-scale language models cannot fully achieve.This may result in unreliable outcomes from the model,lacking basic knowledge or inability to meet specific demands,and may even mislead or offend users.The primary objective of this study is to investigate the neural network methods for style-constrained controllable text generation.To address the challenges of precise control and parameter efficiency in generating controllable text in practical applications,we introduce more sophisticated control mechanisms to adjust the syntax,content,and style of the generated text to produce stylized text that meets specific demands.This paper proposes feasible style-constraint solutions from multiple aspects,including singleattribute control,multi-attribute control,and optimization of controllable sampling in pre-trained large models.Finally,we conduct attribute analysis of the text data and its sociological implications,based on which we propose a stylized resume rewriting method that can also be applied to model debiasing.The main work content and contributions of this paper are summarized as follows:(1)Addressing the exposure bias issue in text generation models and the challenge of balancing creativity with control,this paper introduces a text generation neural network method based on global feedback from reinforcement learning.This approach mitigates the exposure bias issue while enhancing sampling diversity through the integration of semantic feedback,ultimately producing high-quality text that aligns more closely with the target style.Specifically,under supervised training,this method alleviates exposure bias by employing a global discriminator to guide the generation model towards creating outputs that more closely resemble real samples.Simultaneously,the model receives semantic feedback tailored to task characteristics,reinforcing the model’s comprehensive understanding of the semantic relationship between the input and the desired output through semantic soft matching,resulting in higher quality and more creative output.Moreover,the semantic feedback function can be configured differently depending on task characteristics,allowing for flexible and efficient adaptation to various tasks and the generation of stylized samples.Empirical evaluations on classical Chinese translation and couplet generation tasks,both of which require strong language rules,demonstrated the effectiveness of this method in improving the quality and diversity of generated samples.(2)Addressing the complexity of satisfying multiple attribute constraints in multiattribute scenarios,this paper introduces a quadratic programming method for multiattribute text rewriting through neural networks.This method effectively enhances the accuracy of multi-attribute style rewriting while maintaining the quality of the generated text.Specifically,the paper initially undertakes an experimental analysis,positing that the subpar performance of latent variable editing methods under classifier gradient guidance in multi-attribute scenarios may stem from a conflict between the gradient direction and the direction of attribute expression.To rectify this issue,we apply quadratic programming with inequality constraints to correct the gradient,thereby preserving its key attributes.This method,for the first time,considers potential conflicts among different attributes.Grounded in comprehensive mathematical principles,it accurately transitions individual attributes while effectively enhancing the accuracy of multi-attribute transitions.In addition,the method can be utilized as a plug-and-play solution in combination with any autoencoder framework,thus eliminating additional training requirements.Experiments on multi-attribute style transfer in classical style transfer tasks demonstrate that our method not only significantly improves the accuracy of multi-attribute style transfer,but also enhances the quality of the generated text.(3)Addressing the lack of precise control and adaptability in downstream tasks of pre-trained language models,we propose a text style transfer method based on whitening and styling transformation.This method enhances the controllability of the pre-trained language model generation process with minimal computational resource consumption,while also facilitating the semantic preservation of style-independent content during text rewriting.Overall,this approach is parameter-efficient,easily scalable,and provides flexible control over the text transformation.Specifically,this approach constructs a low-rank style subspace with a small amount of additional parameters while the pre-trained model parameters are frozen,through whitening and style transformations.Subsequently,the style-free text is continuously edited in the subspace under specific constraints,achieving controlled style transfer inference.This method allows for the inclusion and combination of constraints in a flexible and fine-grained manner,surpassing prompt-based strategies in terms of precision and control.Furthermore,this method requires learning only a small fraction of parameters(0.1% compared to the GPT2 model),enabling efficient scaling to large-scale language models.Compared to directly modifying the input,this strategy of altering the style subspace expression reduces the risk of major content changes.The effectiveness of this method has been validated through single-attribute and multi-attribute style transfer experiments on commonly used benchmark datasets YELP and GYAFC.(4)Addressing the issue of semantic changes in style-irrelevant content during longtext stylistic rewriting,we propose a method for multiple stylized rewriting of long resume texts based on pre-trained models.This method integrates semantic and syntactic structure information through optimal transport,achieving multi-attribute stylized rewritings such as degendering and formalizing language while enhancing the preservation of original content.It can also function as an attribute debiasing method in language model debiasing tasks.Starting with text data,we systematically analyze and identify the attributes and styles embedded within resume texts.Then uncovers and demonstrates the sociological significance associated with text attributes.These analyses aid in understanding the features of text data attributes and styles,providing support for the design,processing,analysis,and model construction of subsequent text style control tasks.Based on this,facing the challenges of high original content maintenance requirements in resume stylistic rewriting and difficulties in maintaining long-text rewriting content,the original resume’s syntactic structure and semantic information are integrated into the generation process through optimal transport.This enhances the content preservation of long texts under the guidance of semantics and chapter structure,greatly improving the practicality and effectiveness of the method.This method can serve as a means to protect privacy or mitigate potential unintentional discrimination an author might face during job applications,assisting job seekers in more conveniently customizing their resumes.Furthermore,this approach can be applied in language model debiasing tasks.By incorporating constraints from a bias classifier,it generates unbiased text,thereby reducing the potential for generating offensive content that may offend users.Experiments on the resume dataset and Jigsaw dataset have verified the effectiveness of the method.