As it is empirically observed that Vision Transformers (ViTs) are quite
insensitive to the order of input tokens, the need for an appropriate
self-supervised pretext task that enhances the location awareness of ViTs is
becoming evident. To address this, we present DropPos, a novel pretext task
designed to reconstruct Dropped Positions. The formulation of DropPos is
simple: we first drop a large random subset of positional embeddings and then
the model classifies the actual position for each non-overlapping patch among
all possible positions solely based on their visual appearance. To avoid
trivial solutions, we increase the difficulty of this task by keeping only a
subset of patches visible. Additionally, considering there may be different
patches with similar visual appearances, we propose position smoothing and
attentive reconstruction strategies to relax this classification problem, since
it is not necessary to reconstruct their exact positions in these cases.
Empirical evaluations of DropPos show strong capabilities. DropPos outperforms
supervised pre-training and achieves competitive results compared with
state-of-the-art self-supervised alternatives on a wide range of downstream
benchmarks. This suggests that explicitly encouraging spatial reasoning
abilities, as DropPos does, indeed contributes to the improved location
awareness of ViTs. The code is publicly available at
this https URL

对于 Vision Transformers 来说，Droppos 是一种新颖的自我监督任务，它通过增强位置感知能力来提升模型性能。Droppos 通过随机丢弃一部分位置嵌入，根据视觉外观分类具体位置，采用位置平滑和注意力重构策略来解决类别相似性问题，实验结果表明 Droppos 在各类基准测试上表现出色，与现有的自我监督方法相比具有竞争力。