Current state-of-the-art text generators build on powerful language models
such as GPT-2, achieving impressive performance. However, to avoid degenerate
text, they require sampling from a modified softmax, via temperature parameters
or ad-hoc truncation techniques, as in top-$k$ or nucleus sampling. This
creates a mismatch between training and testing conditions. In this paper, we
use the recently introduced entmax transformation to train and sample from a
natively sparse language model, avoiding this mismatch. The result is a text
generator with favorable performance in terms of fluency and consistency, fewer
repetitions, and n-gram diversity closer to human text. In order to evaluate
our model, we propose three new metrics for comparing sparse or truncated
distributions: $\epsilon$-perplexity, sparsemax score, and Jensen-Shannon
divergence. Human-evaluated experiments in story completion and dialogue
generation show that entmax sampling leads to more engaging and coherent
stories and conversations.

本文提出使用 entmax 转换来训练和采样自然稀疏的语言模型，从而避免训练和测试之间的差异，以达到更好的性能和更接近人类文本的 n-gram 多样性，同时提出了三个度量标准来比较稀疏或截断的分布，并通过故事和对话生成等应用展示了该方法的有效性。