Positional encoding plays a crucial role in transformers, significantly
impacting model performance and length generalization. Prior research has
introduced absolute positional encoding (APE) and relative positional encoding
(RPE) to distinguish token positions in given sequences. However, both APE and
RPE remain fixed after model training regardless of input data, limiting their
adaptability and flexibility. Hence, we expect that the desired positional
encoding should be context-adaptive and can be dynamically adjusted with the
given attention. In this paper, we propose a Context-Adaptive Positional
Encoding (CAPE) method, which dynamically and semantically adjusts based on
input context and learned fixed priors. Experimental validation on real-world
datasets (Arxiv, Books3, and CHE) demonstrates that CAPE enhances model
performances in terms of trained length and length generalization, where the
improvements are statistically significant. The model visualization suggests
that our model can keep both local and anti-local information. Finally, we
successfully train the model on sequence length 128 and achieve better
performance at evaluation sequence length 8192, compared with other static
positional encoding methods, revealing the benefit of the adaptive positional
encoding method.

提出了一种动态根据输入语境调整的上下文自适应位置编码方法，通过实验验证在真实世界数据集上改善了模型性能，提升了训练长度和长度泛化能力，同时保留了局部和反局部信息。