We propose a conditional positional encoding (CPE) scheme for vision
Transformers. Unlike previous fixed or learnable positional encodings, which
are pre-defined and independent of input tokens, CPE is dynamically generated
and conditioned on the local neighborhood of the input tokens. As a result, CPE
can easily generalize to the input sequences that are longer than what the
model has ever seen during training. Besides, CPE can keep the desired
translation-invariance in the image classification task, resulting in improved
performance. We implement CPE with a simple Position Encoding Generator (PEG)
to get seamlessly incorporated into the current Transformer framework. Built on
PEG, we present Conditional Position encoding Vision Transformer (CPVT). We
demonstrate that CPVT has visually similar attention maps compared to those
with learned positional encodings and delivers outperforming results. Our code
is available at this https URL .

本文提出了一种条件位置编码方案 (CPE) 用于视觉 Transformers，通过在输入标记的本地邻域上生成并使其动态生成的位置编码，以解决模型寻找较长输入序列的问题，从而在图像分类任务中保持所需的平移不变性，并通过实验证明，CPVT 比使用学习位置编码的注意力地图拥有更好的性能表现。