The standard paradigm of neural language generation adopts maximum likelihood
estimation (MLE) as the optimizing method. From a distributional view, MLE in
fact minimizes the Kullback-Leibler divergence (KLD) between the distribution
of the real data and that of the model. However, this approach forces the model
to distribute non-zero (sometimes large) probability mass to all training
samples regardless of their quality. Moreover, in the attempt to cover the
low-probability regions in the data distribution, the model systematically
overestimates the probability of corrupted text sequences, which we conjecture
is one of the main reasons for text degeneration during autoregressive
decoding. To remedy this problem, we leverage the total variation distance
(TVD) with its robustness to outliers, and develop practical bounds to apply it
to language generation. Then, we introduce the TaiLr objective that balances
the tradeoff of estimating TVD. Intuitively, TaiLr downweights real data
samples that have low model probabilities with tunable penalization intensity.
Experimental results show that our method alleviates the overestimation of
degenerated sequences without sacrificing diversity and improves generation
quality on a wide range of text generation tasks.

采用总变差距离 (TVD) 为目标函数以及 TaiLr 目标函数，可以改善自回归解码期间的文本退化问题，同时在各种文本生成任务中提高生成质量。

在总变差距离下调整语言生成模型

Tailoring Language Generation Models under Total Variation Distance

Recent advances in deep neural language models combined with the capacity of
large scale datasets have accelerated the development of natural language
generation systems that produce fluent and coherent texts (to various degrees
of success) in a multitude of tasks and application contexts. However,
controlling the output of these models for desired user and task needs is still
an open challenge. This is crucial not only to customizing the content and
style of the generated language, but also to their safe and reliable deployment
in the real world. We present an extensive survey on the emerging topic of
constrained neural language generation in which we formally define and
categorize the problems of natural language generation by distinguishing
between conditions and constraints (the latter being testable conditions on the
output text instead of the input), present constrained text generation tasks,
and review existing methods and evaluation metrics for constrained text
generation. Our aim is to highlight recent progress and trends in this emerging
field, informing on the most promising directions and limitations towards
advancing the state-of-the-art of constrained neural language generation
research.

该研究论文系统梳理了存在于神经语言生成技术中的问题，并分别从文本生成任务、方法和评估度量指标三个方面，详细阐述了约束条件和条件约束这两种情况，并总结了当前约束神经语言生成技术的研究进展和面临的主要挑战，以期提高该领域的技术水平。

限制性神经语言生成为什么特别具有挑战性？

Why is constrained neural language generation particularly challenging?

Recent work on abstractive summarization has made progress with neural
encoder-decoder architectures. However, such models are often challenged due to
their lack of explicit semantic modeling of the source document and its
summary. In this paper, we extend previous work on abstractive summarization
using Abstract Meaning Representation (AMR) with a neural language generation
stage which we guide using the source document. We demonstrate that this
guidance improves summarization results by 7.4 and 10.5 points in ROUGE-2 using
gold standard AMR parses and parses obtained from an off-the-shelf parser
respectively. We also find that the summarization performance using the latter
is 2 ROUGE-2 points higher than that of a well-established neural
encoder-decoder approach trained on a larger dataset. Code is available at
https://github.com/sheffieldnlp/AMR2Text-summ

本文提出了通过 Abstract Meaning Representation (AMR) 结合神经语言生成模型的抽象摘要方案，使用源文档进行指导可以提高抽象摘要的质量，并在 ROUGE-2 测试中表现优异。