Understanding the mechanisms behind Vision Transformer (ViT), particularly
its vulnerability to adversarial perturba tions, is crucial for addressing
challenges in its real-world applications. Existing ViT adversarial attackers
rely on la bels to calculate the gradient for perturbation, and exhibit low
transferability to other structures and tasks. In this paper, we present a
label-free white-box attack approach for ViT-based models that exhibits strong
transferability to various black box models, including most ViT variants, CNNs,
and MLPs, even for models developed for other modalities. Our inspira tion
comes from the feature collapse phenomenon in ViTs, where the critical
attention mechanism overly depends on the low-frequency component of features,
causing the features in middle-to-end layers to become increasingly similar and
eventually collapse. We propose the feature diversity attacker to naturally
accelerate this process and achieve remarkable performance and transferability.

提出了一种用于 Vision Transformer 模型的无标签白盒攻击方法，该方法在各种黑盒模型之间表现出较强的迁移性，包括大多数 Vision Transformer 变体、CNN 和 MLP，甚至是针对其他模态开发的模型。攻击的灵感来自于 Vision Transformer 中的特征重叠现象，其中关键的注意力机制过度依赖于特征的低频成分，导致中 - 末层的特征趋向相似并最终崩塌。提出的特征多样性攻击者可以自然加速这一过程，达到卓越的性能和可转移性。

使用特征多样性对抗攻击变形器

Attacking Transformers with Feature Diversity Adversarial Perturbation

We formalize and study a phenomenon called feature collapse that makes
precise the intuitive idea that entities playing a similar role in a learning
task receive similar representations. As feature collapse requires a notion of
task, we leverage a simple but prototypical NLP task to study it. We start by
showing experimentally that feature collapse goes hand in hand with
generalization. We then prove that, in the large sample limit, distinct words
that play identical roles in this NLP task receive identical local feature
representations in a neural network. This analysis reveals the crucial role
that normalization mechanisms, such as LayerNorm, play in feature collapse and
in generalization.

本文研究了一种叫做特征坍塌的现象，即在学习任务中，扮演相似角色的元素会获得类似的表达方式，其中 LayerNorm 等规范化机制在特征坍塌和泛化中扮演着关键作用。通过在 NLP 任务中进行实验，证明了特征坍塌与泛化密切相关，并且在大样本极限下，相同角色的不同单词在神经网络中获得相同的局部特征表达。

特征坍塌

Feature Collapse

Recently, a variety of methods under the name of non-contrastive learning
(like BYOL, SimSiam, SwAV, DINO) show that when equipped with some asymmetric
architectural designs, aligning positive pairs alone is sufficient to attain
good performance in self-supervised visual learning. Despite some
understandings of some specific modules (like the predictor in BYOL), there is
yet no unified theoretical understanding of how these seemingly different
asymmetric designs can all avoid feature collapse, particularly considering
methods that also work without the predictor (like DINO). In this work, we
propose a unified theoretical understanding for existing variants of
non-contrastive learning. Our theory named Rank Differential Mechanism (RDM)
shows that all these asymmetric designs create a consistent rank difference in
their dual-branch output features. This rank difference will provably lead to
an improvement of effective dimensionality and alleviate either complete or
dimensional feature collapse. Different from previous theories, our RDM theory
is applicable to different asymmetric designs (with and without the predictor),
and thus can serve as a unified understanding of existing non-contrastive
learning methods. Besides, our RDM theory also provides practical guidelines
for designing many new non-contrastive variants. We show that these variants
indeed achieve comparable performance to existing methods on benchmark
datasets, and some of them even outperform the baselines. Our code is available
at https://github.com/PKU-ML/Rank-Differential-Mechanism.

本文通过提出 Rank Differential Mechanism (RDM) 理论，对现有的非对比学习方法进行统一理论解释，并为设计许多新的非对比变体提供实用指南。该理论在不同非对称设计方面适用，因此可以作为现有非对比学习方法的统一理解。