When the predicted sequence length exceeds the length seen during training,
the transformer's inference accuracy diminishes. Existing relative position
encoding methods, such as those based on the ALiBi technique, address the
length extrapolation challenge exclusively through the implementation of a
single kernel function, which introduces a constant bias to every post-softmax
attention scores according to their distance. These approaches do not
investigate or employ multiple kernel functions to address the extrapolation
challenge. Drawing on the ALiBi approach, this study proposes a novel relative
positional encoding method, called MEP, which employs a weighted average to
combine distinct kernel functions(such as the exponential kernel and the
Gaussian kernel) to generate a bias that is applied to post-softmax attention
scores. Initially, the framework utilizes various kernel functions to construct
multiple kernel functions. Each kernel function adheres to a consistent mean
weight coefficient, harnessing the synergistic advantages of different kernels
to formulate an innovative bias function. Subsequently, specific slopes are
tailored for each kernel function, applying penalties at varying rates, to
enhance the model's extrapolation capabilities. Finally, this bias is
seamlessly incorporated as a penalty to the post-softmax scores. We present two
distinct versions of our method: a parameter-free variant that requires no new
learnable parameters, which enhances length extrapolation capabilities without
compromising training efficiency, and a parameterized variant capable of
integrating state-of-the-art techniques. Empirical evaluations across diverse
datasets have demonstrated that both variants of our method achieve
state-of-the-art performance, outperforming traditional parameter-free and
parameterized approaches.

当预测序列长度超过训练期间观察到的长度时，transformer 的推断准确性会降低。本研究基于 ALiBi 方法，提出了一种新的相对位置编码方法 MEP，它结合了不同的核函数（如指数核函数和高斯核函数）通过加权平均组成一个偏置，应用于后 softmax 注意力分数，实现了长度推知挑战的解决。实验证明，我们方法的两个版本都达到了最先进的性能，在各种数据集上优于传统的无参数和参数化方法。