Position encoding recently has shown effective in the transformer
architecture. It enables valuable supervision for dependency modeling between
elements at different positions of the sequence. In this paper, we first
investigate various methods to integrate positional information into the
learning process of transformer-based language models. Then, we propose a novel
method named Rotary Position Embedding(RoPE) to effectively leverage the
positional information. Specifically, the proposed RoPE encodes the absolute
position with a rotation matrix and meanwhile incorporates the explicit
relative position dependency in self-attention formulation. Notably, RoPE
enables valuable properties, including the flexibility of sequence length,
decaying inter-token dependency with increasing relative distances, and the
capability of equipping the linear self-attention with relative position
encoding. Finally, we evaluate the enhanced transformer with rotary position
embedding, also called RoFormer, on various long text classification benchmark
datasets. Our experiments show that it consistently overcomes its alternatives.
Furthermore, we provide a theoretical analysis to explain some experimental
results. RoFormer is already integrated into Huggingface:
https://huggingface.co/docs/transformers/model_doc/roformer.

本篇论文研究了在语言模型中如何整合位置信息，并提出了一种名为 RoPE 的方法，它可以将位置信息编码为旋转矩阵，并同时将显式的相对位置依赖性结合到自注意力公式中。实验结果表明，RoPE 使 transformer 在处理长文本分类问题时表现出优越的性能。

RoFormer: 带旋转位置嵌入的增强 Transformer

RoFormer: Enhanced Transformer with Rotary Position Embedding

Position encoding (PE), an essential part of self-attention networks (SANs),
is used to preserve the word order information for natural language processing
tasks, generating fixed position indices for input sequences. However, in
cross-lingual scenarios, e.g. machine translation, the PEs of source and target
sentences are modeled independently. Due to word order divergences in different
languages, modeling the cross-lingual positional relationships might help SANs
tackle this problem. In this paper, we augment SANs with \emph{cross-lingual
position representations} to model the bilingually aware latent structure for
the input sentence. Specifically, we utilize bracketing transduction grammar
(BTG)-based reordering information to encourage SANs to learn bilingual
diagonal alignments. Experimental results on WMT'14 English$\Rightarrow$German,
WAT'17 Japanese$\Rightarrow$English, and WMT'17 Chinese$\Leftrightarrow$English
translation tasks demonstrate that our approach significantly and consistently
improves translation quality over strong baselines. Extensive analyses confirm
that the performance gains come from the cross-lingual information.

本文使用跨语言位置表示法，将自注意力网络中的位置编码扩展为双语感知潜在结构，利用括号转换语法（BTG）基于重排序信息，以鼓励自注意力网络学习双语对角线对齐，并通过 WMT'14 英语⇒德语，WAT'17 日语⇒英语，WMT'17 中英互译任务的实验结果验证，显著且一致地改善了翻译质量。

跨语言位置表示的自注意力

Self-Attention with Cross-Lingual Position Representation

We introduce a new way of learning to encode position information for
non-recurrent models, such as Transformer models. Unlike RNN and LSTM, which
contain inductive bias by loading the input tokens sequentially, non-recurrent
models are less sensitive to position. The main reason is that position
information among input units is not inherently encoded, i.e., the models are
permutation equivalent; this problem justifies why all of the existing models
are accompanied by a sinusoidal encoding/embedding layer at the input. However,
this solution has clear limitations: the sinusoidal encoding is not flexible
enough as it is manually designed and does not contain any learnable
parameters, whereas the position embedding restricts the maximum length of
input sequences. It is thus desirable to design a new position layer that
contains learnable parameters to adjust to different datasets and different
architectures. At the same time, we would also like the encodings to
extrapolate in accordance with the variable length of inputs. In our proposed
solution, we borrow from the recent Neural ODE approach, which may be viewed as
a versatile continuous version of a ResNet. This model is capable of modeling
many kinds of dynamical systems. We model the evolution of encoded results
along position index by such a dynamical system, thereby overcoming the above
limitations of existing methods. We evaluate our new position layers on a
variety of neural machine translation and language understanding tasks, the
experimental results show consistent improvements over the baselines.

提出了一种新的位置信息编码方法，使用神经常微分方法对非循环模型（如 Transformer）进行编码，并证明在翻译和理解任务中，该编码方法与已有编码方法相比具有更好的性能。