Transformers are state of the art models in NLP that map a given input sequence of vectors to an output sequence of vectors. However these models are permutation equivariant, and additive position embeddings to the input are used to supply the information about the order of the input tokens. Further, for some tasks, additional additive segment embeddings are used to denote different types of input sentences. Recent works proposed variations of positional encodings with relative position encodings achieving better performance. In this work, we do a systematic study comparing different position encodings and understanding the reasons for differences in their performance. We demonstrate a simple yet effective way to encode position and segment into the Transformer models. The proposed method performs on par with SOTA on GLUE, XTREME and WMT benchmarks while saving computation costs.

本文介绍了一种新的机制——Decoupled Positional Attention，将位置和段信息编码为Transformer模型，提高了训练和推理效率，在GLUE、XTREME和WMT基准测试中实现了竞争性表现，并进一步将该方法推广到远程的transformers，显示了性能提升。

Transformer 的简单有效位置编码