Transformer language models encode the notion of word order using positional
information. Most commonly, this positional information is represented by
absolute position embeddings (APEs), that are learned from the pretraining
data. However, in natural language, it is not absolute position that matters,
but relative position, and the extent to which APEs can capture this type of
information has not been investigated. In this work, we observe that models
trained with APE over-rely on positional information to the point that they
break-down when subjected to sentences with shifted position information.
Specifically, when models are subjected to sentences starting from a non-zero
position (excluding the effect of priming), they exhibit noticeably degraded
performance on zero to full-shot tasks, across a range of model families and
model sizes. Our findings raise questions about the efficacy of APEs to model
the relativity of position information, and invite further introspection on the
sentence and word order processing strategies employed by these models.

使用 Transformer 语言模型中的绝对位置嵌入会导致对相对位置信息的过度依赖，从而使得模型在句子位置信息被移位时性能显著下降，这表明了使用绝对位置嵌入所建模 “相对位置信息” 的功效需要重新审视。