Transformers with powerful global relation modeling abilities have been
introduced to fundamental computer vision tasks recently. As a typical example,
the Vision Transformer (ViT) directly applies a pure transformer architecture
on image classification, by simply splitting images into tokens with a fixed
length, and employing transformers to learn relations between these tokens.
However, such naive tokenization could destruct object structures, assign grids
to uninterested regions such as background, and introduce interference signals.
To mitigate the above issues, in this paper, we propose an iterative and
progressive sampling strategy to locate discriminative regions. At each
iteration, embeddings of the current sampling step are fed into a transformer
encoder layer, and a group of sampling offsets is predicted to update the
sampling locations for the next step. The progressive sampling is
differentiable. When combined with the Vision Transformer, the obtained PS-ViT
network can adaptively learn where to look. The proposed PS-ViT is both
effective and efficient. When trained from scratch on ImageNet, PS-ViT performs
3.8% higher than the vanilla ViT in terms of top-1 accuracy with about
$4\times$ fewer parameters and $10\times$ fewer FLOPs. Code is available at
this https URL

本研究提出一种迭代和渐进式采样策略，以定位具有区分性的区域，并与 Vision Transformer 结合起来，形成 PS-ViT 网络。该网络可自适应地学习何时观察图像的哪些区域，从而在 ImageNet 数据集上表现出比原始 ViT 网络高 3.8％的 top-1 准确性（使用约 4 倍的参数和 10 倍的运算次数）。