Recent advances in Transformer models allow for unprecedented sequence
lengths, due to linear space and time complexity. In the meantime, relative
positional encoding (RPE) was proposed as beneficial for classical Transformers
and consists in exploiting lags instead of absolute positions for inference.
Still, RPE is not available for the recent linear-variants of the Transformer,
because it requires the explicit computation of the attention matrix, which is
precisely what is avoided by such methods. In this paper, we bridge this gap
and present Stochastic Positional Encoding as a way to generate PE that can be
used as a replacement to the classical additive (sinusoidal) PE and provably
behaves like RPE. The main theoretical contribution is to make a connection
between positional encoding and cross-covariance structures of correlated
Gaussian processes. We illustrate the performance of our approach on the
Long-Range Arena benchmark and on music generation.

本研究提出了随机位置编码的方法来生成代替传统加性（正弦）位置编码的 PE，并证明其类似于 RPE，建立了位置编码与相关高斯过程的交叉协方差结构之间的联系。这种方法能够弥补针对最近的线性 Transformer 变量不可用于 RPE 的问题，并展示了其在 Long-Range Arena 基准测试和音乐生成方面的性能表现。