Keyword-Constrained Text Generation

This research focuses on text generation based on keyword constraints,which introduces keywords as additional knowledge constraints to control the output of language models.Compared with conditional variables based on random sample vectors,keywords have better interpretability and can be easily obtained from users and other upstream applications.On the one hand,we introduce two different types of keywords,pattern and content,by fine-tuning a small language model.The pattern types include but are not limited to style,personality,emotion,age,and gender.Content can be any keyword that needs to be added to the text.In order to better verify the text generation effect,we design a two-stage response generation framework,introducing pattern control in the generation stage and adding content keywords in the modification stage to help the model generate responses with diverse patterns and controllable content.This method can be applied to various practical scenarios,such as daily conversations,product and news comments,etc.Our dataset and code are available at GitHub③.On the other hand,we introduce task keywords for generating prompt text for large language models to improve their evaluation performance on downstream tasks.Large language models(LLMs)are popular due to their outstanding abilities.However,fine-tuning a specific LLM or engineering task-specific prompts may negatively impact their generalization abilities.To address this,we introduce UPRISE,which automatically retrieves prompts for a given task input by tuning a lightweight and versatile retriever.We demonstrate the universality of the method across tasks and models,as the retriever is tuned on multiple tasks but tested on unseen task types.We tune the retriever on a relatively small LLM GPT-Neo-2.7B but test it on larger and different LLMs(such as BLOOM-7.1B,OPT-66B,and GPT3-175B).Additionally,experiments show that UPRISE can alleviate the hallucination problem of ChatGPT,indicating its potential to improve even the strongest LLMs.Our model and code are available at Github.