Inspired by the Bloch Sphere representation, we propose a novel rotary
position encoding on a three-dimensional sphere, named 3D Rotary Position
Encoding (3D-RPE). 3D-RPE is an advanced version of the widely used 2D Rotary
Position Encoding (RoPE), with two major advantages for modeling long contexts:
controllable long-term decay and improved position resolution. For controllable
long-term decay, 3D-RPE allows for the regulation of long-term decay within the
chunk size, ensuring the modeling of relative positional information between
tokens at a distant relative position. For enhanced position resolution, 3D-RPE
can mitigate the degradation of position resolution caused by position
interpolation on RoPE. We have conducted experiments on long-context Natural
Language Understanding (NLU) and long-sequence Language Modeling (LM) tasks.
From the experimental results, 3D-RPE achieved performance improvements over
RoPE, especially in long-context NLU tasks.

使用 3D 旋转位置编码（3D-RPE）在三维球上对象进行编码，具有可控的长期衰减和提高的位置分辨率，可应用于模拟长上下文和长序列任务中，特别在长上下文自然语言理解任务中表现出性能的提升。