Steering language generation towards objectives or away from undesired
content has been a long-standing goal in utilizing language models (LM). Recent
work has demonstrated reinforcement learning and weighted decoding as effective
approaches to achieve a higher level of language control and quality with pros
and cons. In this work, we propose a novel critic decoding method for
controlled language generation (CriticControl) that combines the strengths of
reinforcement learning and weighted decoding. Specifically, we adopt the
actor-critic framework to train an LM-steering critic from non-differentiable
reward models. And similar to weighted decoding, our method freezes the
language model and manipulates the output token distribution using called
critic, improving training efficiency and stability. Evaluation of our method
on three controlled generation tasks, namely topic control, sentiment control,
and detoxification, shows that our approach generates more coherent and
well-controlled texts than previous methods. In addition, CriticControl
demonstrates superior generalization ability in zero-shot settings. Human
evaluation studies also corroborate our findings.

本文介绍了一种新型的批判性解码方法来实现语言生成的控制，在控制主题、情感和排毒方面，表现得比以前的方法更好，并且在零 - shot 情况下表现出卓越的泛化能力。

基于评论引导的控制文本生成解码

Critic-Guided Decoding for Controlled Text Generation

Large-scale pre-trained language models have achieved great success on
natural language generation tasks. However, it is difficult to control the
pre-trained language models to generate sentences with the desired attribute
such as topic and sentiment, etc. Recently, Bayesian Controllable Language
Models (BCLMs) have been shown to be efficient in controllable language
generation. Rather than fine-tuning the parameters of pre-trained language
models, BCLMs use external discriminators to guide the generation of
pre-trained language models. However, the mismatch between training and
inference of BCLMs limits the performance of the models. To address the
problem, in this work we propose a "Gemini Discriminator" for controllable
language generation which alleviates the mismatch problem with a small
computational cost. We tested our method on two controllable language
generation tasks: sentiment control and topic control. On both tasks, our
method reached achieved new state-of-the-art results in automatic and human
evaluations.

本文提出一种控制语言生成的技术 Gemini Discriminator，有效解决了 Bayesian Controllable Language Models 中的训练和推断不匹配的问题，并在情感和主题控制任务中取得了最新的最优结果。