Relative positional embeddings (RPE) have received considerable attention
since RPEs effectively model the relative distance among tokens and enable
length extrapolation. We propose KERPLE, a framework that generalizes relative
position embedding for extrapolation by kernelizing positional differences. We
achieve this goal using conditionally positive definite (CPD) kernels, a class
of functions known for generalizing distance metrics. To maintain the inner
product interpretation of self-attention, we show that a CPD kernel can be
transformed into a PD kernel by adding a constant offset. This offset is
implicitly absorbed in the Softmax normalization during self-attention. The
diversity of CPD kernels allows us to derive various RPEs that enable length
extrapolation in a principled way. Experiments demonstrate that the logarithmic
variant achieves excellent extrapolation performance on three large language
modeling datasets. Our implementation and pretrained checkpoints are released
at this https URL.

提出了一种广义的相对位置嵌入的框架 KERPLE，通过核化位置差异实现外推，实验结果表明，对于三个大型语言模型数据集，采用对数变体可以实现良好的外推性能。