The attention mechanism is a critical component of Large Language Models
(LLMs) that allows tokens in a sequence to interact with each other, but is
order-invariant. Incorporating position encoding (PE) makes it possible to
address by position, such as attending to the i-th token. However, current PE
methods use token counts to derive position, and thus cannot generalize to
higher levels of abstraction, such as attending to the i-th sentence. In this
paper, we propose a new position encoding method, Contextual Position Encoding
(CoPE), that allows positions to be conditioned on context by incrementing
position only on certain tokens determined by the model. This allows more
general position addressing such as attending to the $i$-th particular word,
noun, or sentence. We show that CoPE can solve the selective copy, counting and
Flip-Flop tasks where popular position embeddings fail, and improves perplexity
on language modeling and coding tasks.

提出了一种新的位置编码方法：上下文位置编码 (CoPE)，使得位置可以被上下文条件化，可以实现更一般的位置指定，例如参考第 $i$ 个特定词、名词或句子。该方法能够解决选择性复制、计数和翻转任务，并提高语言建模和编码任务的困惑度。