Mechanisms for encoding positional information are central for
transformer-based language models. In this paper, we analyze the position
embeddings of existing language models, finding strong evidence of translation
invariance, both for the embeddings themselves and for their effect on
self-attention. The degree of translation invariance increases during training
and correlates positively with model performance. Our findings lead us to
propose translation-invariant self-attention (TISA), which accounts for the
relative position between tokens in an interpretable fashion without needing
conventional position embeddings. Our proposal has several theoretical
advantages over existing position-representation approaches. Experiments show
that it improves on regular ALBERT on GLUE tasks, while only adding orders of
magnitude less positional parameters.

本文分析了现有语言模型的位置嵌入，发现其对于自注意力有着强烈的翻译不变性，并提出了一种解决方案 —— 翻译不变的自注意力模型（TISA），它可以以一种可解释的方式考虑标记之间的相对位置，而无需传统的位置嵌入，实验表明它在 GLUE 任务上的性能优于 ALBERT 模型。