Transformer architectures rely on explicit position encodings in order to
preserve a notion of word order. In this paper, we argue that existing work
does not fully utilize position information. For example, the initial proposal
of a sinusoid embedding is fixed and not learnable. In this paper, we first
review absolute position embeddings and existing methods for relative position
embeddings. We then propose new techniques that encourage increased interaction
between query, key and relative position embeddings in the self-attention
mechanism. Our most promising approach is a generalization of the absolute
position embedding, improving results on SQuAD1.1 compared to previous position
embeddings approaches. In addition, we address the inductive property of
whether a position embedding can be robust enough to handle long sequences. We
demonstrate empirically that our relative position embedding method is
reasonably generalized and robust from the inductive perspective. Finally, we
show that our proposed method can be adopted as a near drop-in replacement for
improving the accuracy of large models with a small computational budget.

本文提出基于自我关注机制的绝对位置嵌入和相对位置嵌入方法，通过增加查询、键和相对位置嵌入之间的交互，进一步优化了位置信息的利用。其最有前途的一种方法是将绝对位置嵌入泛化，相比之前的位置嵌入方法，在 SQuAD1.1 上有更好的表现。本文还通过实验证明了相对位置嵌入方法具有合理的泛化性和鲁棒性。最后，还展示了这种新方法可以用于在小的计算预算下提高大型模型的准确性。