Large pre-trained language models have repeatedly shown their ability to
produce fluent text. Yet even when starting from a prompt, generation can
continue in many plausible directions. Current decoding methods with the goal
of controlling generation, e.g., to ensure specific words are included, either
require additional models or fine-tuning, or work poorly when the task at hand
is semantically unconstrained, e.g., story generation. In this work, we present
a plug-and-play decoding method for controlled language generation that is so
simple and intuitive, it can be described in a single sentence: given a topic
or keyword, we add a shift to the probability distribution over our vocabulary
towards semantically similar words. We show how annealing this distribution can
be used to impose hard constraints on language generation, something no other
plug-and-play method is currently able to do with SOTA language generators.
Despite the simplicity of this approach, we see it works incredibly well in
practice: decoding from GPT-2 leads to diverse and fluent sentences while
guaranteeing the appearance of given guide words. We perform two user studies,
revealing that (1) our method outperforms competing methods in human
evaluations; and (2) forcing the guide words to appear in the generated text
has no impact on the fluency of the generated text.

通过向语料库单词的概率分布添加位移来控制语言生成，能保证给定的关键词出现，相比类似方法具有更高的效果。

一种插拔式控制文本生成方法

A Plug-and-Play Method for Controlled Text Generation

While data augmentation is an important trick to boost the accuracy of deep
learning methods in computer vision tasks, its study in natural language tasks
is still very limited. In this paper, we present a novel data augmentation
method for neural machine translation. Different from previous augmentation
methods that randomly drop, swap or replace words with other words in a
sentence, we softly augment a randomly chosen word in a sentence by its
contextual mixture of multiple related words. More accurately, we replace the
one-hot representation of a word by a distribution (provided by a language
model) over the vocabulary, i.e., replacing the embedding of this word by a
weighted combination of multiple semantically similar words. Since the weights
of those words depend on the contextual information of the word to be replaced,
the newly generated sentences capture much richer information than previous
augmentation methods. Experimental results on both small scale and large scale
machine translation datasets demonstrate the superiority of our method over
strong baselines.

本文提出了一种新颖的神经机器翻译数据增强方法，即通过语言模型提供的词汇分布来增强句子中的特定单词，从而比之前的增强方法更好地捕捉上下文语义信息。实验结果表明该方法在小规模和大规模机器翻译数据集上都优于强基线。