Transformer models are permutation equivariant. To supply the order and type
information of the input tokens, position and segment embeddings are usually
added to the input. Recent works proposed variations of positional encodings
with relative position encodings achieving better performance. Our analysis
shows that the gain actually comes from moving positional information to
attention layer from the input. Motivated by this, we introduce Decoupled
Positional Attention for Transformers (DIET), a simple yet effective mechanism
to encode position and segment information into the Transformer models. The
proposed method has faster training and inference time, while achieving
competitive performance on GLUE, XTREME and WMT benchmarks. We further
generalize our method to long-range transformers and show performance gain.

本文介绍了一种新的机制 ——Decoupled Positional Attention，将位置和段信息编码为 Transformer 模型，提高了训练和推理效率，在 GLUE、XTREME 和 WMT 基准测试中实现了竞争性表现，并进一步将该方法推广到远程的 transformers，显示了性能提升。