While large language models (LLMs) have achieved impressive performance in
generating fluent and realistic text, controlling the generated text so that it
exhibits properties such as safety, factuality, and non-toxicity remains
challenging. % such as DExperts, GeDi, and rectification Existing
decoding-based methods are static in terms of the dimension of control; if the
target subject is changed, they require new training. Moreover, it can quickly
become prohibitive to concurrently control multiple subjects. In this work, we
introduce SF-GEN, which is grounded in two primary concepts: successor features
(SFs) to decouple the LLM's dynamics from task-specific rewards, and language
model rectification to proportionally adjust the probability of selecting a
token based on the likelihood that the finished text becomes undesired. SF-GEN
seamlessly integrates the two to enable dynamic steering of text generation
with no need to alter the LLM's parameters. Thanks to the decoupling effect
induced by successor features, our method proves to be memory-wise and
computationally efficient for training as well as decoding, especially when
dealing with multiple target subjects. To the best of our knowledge, our
research represents the first application of successor features in text
generation. In addition to its computational efficiency, the resultant language
produced by our method is comparable to the SOTA (and outperforms baselines) in
both control measures as well as language quality, which we demonstrate through
a series of experiments in various controllable text generation tasks.

大型语言模型（LLMs）在生成流畅和逼真的文本方面取得了令人印象深刻的性能，但控制生成的文本以展现安全性、真实性和无毒性等属性仍然具有挑战性。本研究提出了 SF-GEN，它基于两个主要概念：后继特征（SFs）用于将 LLM 的动态与任务特定奖励解耦，以及语言模型纠正以根据完成文本变为不希望的概率比例调整选择标记的概率。SF-GEN 无需改变 LLM 的参数即可实现对文本生成的动态导向，是一种在训练和解码过程中在内存和计算效率上都表现出色的方法，特别适用于处理多个目标主题。同时，我们的方法生成的语言质量与 SOTA（并且优于基准）在控制措施和语言质量方面相当，我们通过一系列各种可控文本生成任务的实验证明了这一点。