Text generation is ubiquitous in many NLP tasks, from summarization, to
dialogue and machine translation. The dominant parametric approach is based on
locally normalized models which predict one word at a time. While these work
remarkably well, they are plagued by exposure bias due to the greedy nature of
the generation process. In this work, we investigate un-normalized energy-based
models (EBMs) which operate not at the token but at the sequence level. In
order to make training tractable, we first work in the residual of a pretrained
locally normalized language model and second we train using noise contrastive
estimation. Furthermore, since the EBM works at the sequence level, we can
leverage pretrained bi-directional contextual representations, such as BERT and
RoBERTa. Our experiments on two large language modeling datasets show that
residual EBMs yield lower perplexity compared to locally normalized baselines.
Moreover, generation via importance sampling is very efficient and of higher
quality than the baseline models according to human evaluation.

本文探讨了基于序列级别的未标准化的能量模型应用于文本生成，通过在预训练的局部标准化语言模型的残差范围内工作，结合噪声对比估计来训练，同时利用 BERT 和 RoBERTa 等预先训练的双向上下文表示，结果表明在二个大型语言建模数据集上，残差 EBM 相对于局部标准化基线具有更低的困惑度，并且通过重要性抽样生成的效率比基线模型更高，并且在人类评估中具有更高的生成质量。