Recently, vision Transformers (ViTs) have been actively applied to
fine-grained visual recognition (FGVR). ViT can effectively model the
interdependencies between patch-divided object regions through an inherent
self-attention mechanism. In addition, patch selection is used with ViT to
remove redundant patch information and highlight the most discriminative object
patches. However, existing ViT-based FGVR models are limited to single-scale
processing, and their fixed receptive fields hinder representational richness
and exacerbate vulnerability to scale variability. Therefore, we propose
multi-scale patch selection (MSPS) to improve the multi-scale capabilities of
existing ViT-based models. Specifically, MSPS selects salient patches of
different scales at different stages of a multi-scale vision Transformer
(MS-ViT). In addition, we introduce class token transfer (CTT) and multi-scale
cross-attention (MSCA) to model cross-scale interactions between selected
multi-scale patches and fully reflect them in model decisions. Compared to
previous single-scale patch selection (SSPS), our proposed MSPS encourages
richer object representations based on feature hierarchy and consistently
improves performance from small-sized to large-sized objects. As a result, we
propose M2Former, which outperforms CNN-/ViT-based models on several widely
used FGVR benchmarks.

提出了一种多尺度补丁选择 (MSPS) 方法，以改善现有基于 Vision Transformers 的模型的多尺度能力。通过在多尺度 Vision Transformer 上的不同阶段选择不同尺度的显著补丁，并引入类令牌传递 (CTT) 和多尺度交叉注意力 (MSCA)，以模拟选定的多尺度补丁之间的交叉尺度相互作用并完全反映在模型决策中，从而提高了特征分层的丰富物体表征并在多个广泛使用的细粒度视觉识别基准上优于 CNN-/ViT- 基模型的 M2Former。