In Transformer-based architectures, the attention mechanism is inherently
permutation-invariant with respect to the input sequence's tokens. To impose
sequential order, token positions are typically encoded using a scheme with
either fixed or learnable parameters. We introduce Hyperbolic Positional
Encoding (HyPE), a novel method that utilizes hyperbolic functions' properties
to encode tokens' relative positions. This approach biases the attention
mechanism without the necessity of storing the $O(L^2)$ values of the mask,
with $L$ being the length of the input sequence. HyPE leverages preliminary
concatenation operations and matrix multiplications, facilitating the encoding
of relative distances indirectly incorporating biases into the softmax
computation. This design ensures compatibility with FlashAttention-2 and
supports the gradient backpropagation for any potential learnable parameters
within the encoding. We analytically demonstrate that, by careful
hyperparameter selection, HyPE can approximate the attention bias of ALiBi,
thereby offering promising generalization capabilities for contexts extending
beyond the lengths encountered during pretraining. The experimental evaluation
of HyPE is proposed as a direction for future research.

通过引入超几何位置编码（HyPE），本研究提出一种新颖方法，利用超几何函数的特性来编码令牌的相对位置，以实现对令牌顺序的约束。该方法通过初步的连接操作和矩阵乘法实现相对距离的编码，间接地将偏置引入到 softmax 计算中，从而确保了与 FlashAttention-2 的兼容性，并支持对编码中任何潜在可学习参数的梯度反向传播。通过仔细的超参数选择，HyPE 可以近似于 ALiBi 的注意力偏置，从而在预训练期间具有良好的泛化能力，为超出预训练长度的上下文提供了有前途的泛化能力。对 HyPE 的实验评估可作为未来研究的一个方向。

HyPE：带有双曲偏差的相对位置编码的注意力

HyPE: Attention with Hyperbolic Biases for Relative Positional Encoding

Relying entirely on an attention mechanism, the Transformer introduced by
Vaswani et al. (2017) achieves state-of-the-art results for machine
translation. In contrast to recurrent and convolutional neural networks, it
does not explicitly model relative or absolute position information in its
structure. Instead, it requires adding representations of absolute positions to
its inputs. In this work we present an alternative approach, extending the
self-attention mechanism to efficiently consider representations of the
relative positions, or distances between sequence elements. On the WMT 2014
English-to-German and English-to-French translation tasks, this approach yields
improvements of 1.3 BLEU and 0.3 BLEU over absolute position representations,
respectively. Notably, we observe that combining relative and absolute position
representations yields no further improvement in translation quality. We
describe an efficient implementation of our method and cast it as an instance
of relation-aware self-attention mechanisms that can generalize to arbitrary
graph-labeled inputs.

本研究通过在自我关注机制中引入对序列元素相对位置或距离的表示，比绝对位置表示获得了更好的机器翻译结果。同时，将相对位置表示和绝对位置表示相结合并不能进一步提高翻译质量。