Steering language generation towards objectives or away from undesired
content has been a long-standing goal in utilizing language models (LM). Recent
work has demonstrated reinforcement learning and weighted decoding as effective
approaches to achieve a higher level of language control and quality with pros
and cons. In this work, we propose a novel critic decoding method for
controlled language generation (CriticControl) that combines the strengths of
reinforcement learning and weighted decoding. Specifically, we adopt the
actor-critic framework to train an LM-steering critic from non-differentiable
reward models. And similar to weighted decoding, our method freezes the
language model and manipulates the output token distribution using called
critic, improving training efficiency and stability. Evaluation of our method
on three controlled generation tasks, namely topic control, sentiment control,
and detoxification, shows that our approach generates more coherent and
well-controlled texts than previous methods. In addition, CriticControl
demonstrates superior generalization ability in zero-shot settings. Human
evaluation studies also corroborate our findings.

本文介绍了一种新型的批判性解码方法来实现语言生成的控制，在控制主题、情感和排毒方面，表现得比以前的方法更好，并且在零 - shot 情况下表现出卓越的泛化能力。

基于评论引导的控制文本生成解码

Critic-Guided Decoding for Controlled Text Generation

Large-scale pre-trained language models have achieved great success on
natural language generation tasks. However, it is difficult to control the
pre-trained language models to generate sentences with the desired attribute
such as topic and sentiment, etc. Recently, Bayesian Controllable Language
Models (BCLMs) have been shown to be efficient in controllable language
generation. Rather than fine-tuning the parameters of pre-trained language
models, BCLMs use external discriminators to guide the generation of
pre-trained language models. However, the mismatch between training and
inference of BCLMs limits the performance of the models. To address the
problem, in this work we propose a "Gemini Discriminator" for controllable
language generation which alleviates the mismatch problem with a small
computational cost. We tested our method on two controllable language
generation tasks: sentiment control and topic control. On both tasks, our
method reached achieved new state-of-the-art results in automatic and human
evaluations.

本文提出一种控制语言生成的技术 Gemini Discriminator，有效解决了 Bayesian Controllable Language Models 中的训练和推断不匹配的问题，并在情感和主题控制任务中取得了最新的最优结果。

贝叶斯可控语言模型的训练和推理之间的桥梁

Bridging the Gap Between Training and Inference of Bayesian Controllable Language Models

This paper proposes a simple method for controllable text generation based on
weighting logits with a free-form classifier, namely CAIF sampling. Using an
arbitrary text classifier, we adjust a small part of a language model's logits
and guide text generation towards or away from classifier prediction. We
experimented with toxicity avoidance and sentiment control tasks and showed
that the proposed method significantly outperforms recent PPLM, GeDi, and
DExperts on PPL and task accuracy metrics based on the external classifier of
generated texts. In addition, compared to other approaches, it is easier to
implement and tune and has significantly fewer restrictions and requirements.

本研究提出了一种基于分类器的自由形式抽样（CAIF sampling）的可控文本生成方法，在可用的分类器权重下调整语言模型的 logits，从而引导文本生成向预测的分类器结果或远离分类器结果，实验表明该方法在毒性避免和情感控制方面优于最近的 PPLM、GeDi 和 DExperts，并且相对于其他方法，更易于实现和调整，并且要求更少的限制和要求。