High-resolution images offer more information about scenes that can improve
model accuracy. However, the dominant model architecture in computer vision,
the vision transformer (ViT), cannot effectively leverage larger images without
finetuning -- ViTs poorly extrapolate to more patches at test time, although
transformers offer sequence length flexibility. We attribute this shortcoming
to the current patch position encoding methods, which create a distribution
shift when extrapolating.
We propose a drop-in replacement for the position encoding of plain ViTs that
restricts attention heads to fixed fields of view, pointed in different
directions, using 2D attention masks. Our novel method, called LookHere,
provides translation-equivariance, ensures attention head diversity, and limits
the distribution shift that attention heads face when extrapolating. We
demonstrate that LookHere improves performance on classification (avg. 1.6%),
against adversarial attack (avg. 5.4%), and decreases calibration error (avg.
1.5%) -- on ImageNet without extrapolation. With extrapolation, LookHere
outperforms the current SoTA position encoding method, 2D-RoPE, by 21.7% on
ImageNet when trained at $224^2$ px and tested at $1024^2$ px. Additionally, we
release a high-resolution test set to improve the evaluation of high-resolution
image classifiers, called ImageNet-HR.

我们提出了一种名为 LookHere 的新方法，用于限制 2D 注意力掩码中的固定视野的注意力头，提供平移等变性，确保注意力头的多样性，并限制注意力头在外推时面临的分布偏移，从而改善图像分类、对抗攻击和校准误差的性能。该方法在 ImageNet 数据集上进行了测试，结果显示 LookHere 在高分辨率图像分类任务中表现优于当前位置编码方法 2D-RoPE。