Length extrapolation is a desirable property that permits training a
transformer language model on short sequences and retaining similar
perplexities when the model is tested on substantially longer sequences. A
relative positional embedding mechanism applied on the transformer
self-attention matrix, ALiBi, demonstrates the length extrapolation property
with the widest usage to date. In this paper, we show that ALiBi surprisingly
does not utilize tokens further than the training sequence length, which can be
explained by its implicit windowed attention effect that aligns the receptive
field during training and testing stages. Inspired by ALiBi and the receptive
filed alignment hypothesis, we propose another transformer positional embedding
design named~\textbf{Sandwich} that uses longer than training sequence length
information, and it is a greatly simplified formulation of the earliest
proposed Sinusoidal positional embedding. Finally, we show that both ALiBi and
Sandwich enable efficient inference thanks to their implicit windowed attention
effect.

研究了相对位置嵌入在语言模型上的应用，提出了基于对齐假设的自注意力机制，在训练过程中对齐输入，在测试过程中保证了相对位置嵌入的性质。提出的 Sandwich positional embedding 将比训练序列更长的信息融入模型之中，且由于隐式窗口化的自注意力机制，其可实现高效的推断。

感受野对齐实现 Transformer 长度外推

Receptive Field Alignment Enables Transformer Length Extrapolation

Transformer has shown promising results in many sequence to sequence
transformation tasks recently. It utilizes a number of feed-forward
self-attention layers to replace the recurrent neural networks (RNN) in
attention-based encoder decoder (AED) architecture. Self-attention layer learns
temporal dependence by incorporating sinusoidal positional embedding of tokens
in a sequence for parallel computing. Quicker iteration speed in training than
sequential operation of RNN can be obtained. Deeper layers of the transformer
also make it perform better than RNN-based AED. However, this parallelization
ability is lost when applying scheduled sampling training. Self-attention with
sinusoidal positional embedding may cause performance degradations for longer
sequences that have similar acoustic or semantic information at different
positions as well. To address these problems, we propose to use parallel
scheduled sampling (PSS) and relative positional embedding (RPE) to help the
transformer generalize to unseen data. Our proposed methods achieve a 7%
relative improvement for short utterances and a 70% relative gain for long
utterances on a 10,000-hour Mandarin ASR task.

本文中，我们提出了一种采用 parallel scheduling sampling (PSS) 和 relative positional embedding (RPE) 来帮助 Transformer 泛化到看不见的数据的方法。我们的方法在 10,000 小时普通话自动语音识别任务上，对于短语音有 7% 的相对改进和对于长语音有 70% 的相对改进。