The input tokens to Vision Transformers carry little semantic meaning as they
are defined as regular equal-sized patches of the input image, regardless of
its content. However, processing uniform background areas of an image should
not necessitate as much compute as dense, cluttered areas. To address this
issue, we propose a dynamic mixed-scale tokenization scheme for ViT, MSViT. Our
method introduces a conditional gating mechanism that selects the optimal token
scale for every image region, such that the number of tokens is dynamically
determined per input. The proposed gating module is lightweight, agnostic to
the choice of transformer backbone, and trained within a few epochs (e.g., 20
epochs on ImageNet) with little training overhead. In addition, to enhance the
conditional behavior of the gate during training, we introduce a novel
generalization of the batch-shaping loss. We show that our gating module is
able to learn meaningful semantics despite operating locally at the coarse
patch-level. We validate MSViT on the tasks of classification and segmentation
where it leads to improved accuracy-complexity trade-off.

本文提出了一种动态混合尺度的 Vision Transformers 模式，通过引入条件门控机制，智能选择每个图像区域的最佳令牌规模进行处理，从而提高图像分类的精度和效率。