Softmax is the de facto standard in modern neural networks for language
processing when it comes to normalizing logits. However, by producing a dense
probability distribution each token in the vocabulary has a nonzero chance of
being selected at each generation step, leading to a variety of reported
problems in text generation. $\alpha$-entmax of Peters et al. (2019,
arXiv:1905.05702) solves this problem, but is considerably slower than softmax.
In this paper, we propose an alternative to $\alpha$-entmax, which keeps its
virtuous characteristics, but is as fast as optimized softmax and achieves on
par or better performance in machine translation task.

本文研究了针对语言处理中神经网络常用的 softmax 在文本生成中出现的问题，提出了一种替代 alpha-entmax 的方法，并在机器翻译任务中取得了与 alpha-entmax 相当或更好的性能。

加速 Entmax

Speeding Up Entmax

Exponential families are widely used in machine learning; they include many
distributions in continuous and discrete domains (e.g., Gaussian, Dirichlet,
Poisson, and categorical distributions via the softmax transformation).
Distributions in each of these families have fixed support. In contrast, for
finite domains, there has been recent work on sparse alternatives to softmax
(e.g. sparsemax and alpha-entmax), which have varying support, being able to
assign zero probability to irrelevant categories. This paper expands that work
in two directions: first, we extend alpha-entmax to continuous domains,
revealing a link with Tsallis statistics and deformed exponential families.
Second, we introduce continuous-domain attention mechanisms, deriving efficient
gradient backpropagation algorithms for alpha in {1,2}. Experiments on
attention-based text classification, machine translation, and visual question
answering illustrate the use of continuous attention in 1D and 2D, showing that
it allows attending to time intervals and compact regions.

本文将 alpha-entmax 方法扩展到连续域，同时提出了基于连续域注意机制的梯度反向传播算法，并通过在文本分类、机器翻译和视觉问答方面的实验表明了连续注意力机制的应用。

稀疏和连续的注意机制

Sparse and Continuous Attention Mechanisms

Attention mechanisms have become ubiquitous in NLP. Recent architectures,
notably the Transformer, learn powerful context-aware word representations
through layered, multi-headed attention. The multiple heads learn diverse types
of word relationships. However, with standard softmax attention, all attention
heads are dense, assigning a non-zero weight to all context words. In this
work, we introduce the adaptively sparse Transformer, wherein attention heads
have flexible, context-dependent sparsity patterns. This sparsity is
accomplished by replacing softmax with $\alpha$-entmax: a differentiable
generalization of softmax that allows low-scoring words to receive precisely
zero weight. Moreover, we derive a method to automatically learn the $\alpha$
parameter -- which controls the shape and sparsity of $\alpha$-entmax --
allowing attention heads to choose between focused or spread-out behavior. Our
adaptively sparse Transformer improves interpretability and head diversity when
compared to softmax Transformers on machine translation datasets. Findings of
the quantitative and qualitative analysis of our approach include that heads in
different layers learn different sparsity preferences and tend to be more
diverse in their attention distributions than softmax Transformers.
Furthermore, at no cost in accuracy, sparsity in attention heads helps to
uncover different head specializations.

介绍了自适应稀疏转换器，使用 α-entmax 代替 softmax，可以适应性地学习头部偏爱的结构，并改善了 softmax 转换器在机器翻译数据集上的可解释性和头部多样性。