Vision transformers (ViTs) have demonstrated remarkable performance in a
variety of vision tasks. Despite their promising capabilities, training a ViT
requires a large amount of diverse data. Several studies empirically found that
using rich data augmentations, such as Mixup, Cutmix, and random erasing, is
critical to the successful training of ViTs. Now, the use of rich data
augmentations has become a standard practice in the current state. However, we
report a vulnerability to this practice: Certain data augmentations such as
Mixup cause a variance shift in the positional embedding of ViT, which has been
a hidden factor that degrades the performance of ViT during the test phase. We
claim that achieving a stable effect from positional embedding requires a
specific condition on the image, which is often broken for the current data
augmentation methods. We provide a detailed analysis of this problem as well as
the correct configuration for these data augmentations to remove the side
effects of variance shift. Experiments showed that adopting our guidelines
improves the performance of ViTs compared with the current configuration of
data augmentations.

我们提供了关于 Vision Transformers 中数据增强的问题以及正确的配置，以消除方差偏移的副作用，并通过实验证明采用我们的准则与当前数据增强配置相比能够提高 Vision Transformers 的性能。

配置数据增强以减少视觉 Transformer 中位置嵌入的方差偏移

Configuring Data Augmentations to Reduce Variance Shift in Positional  Embedding of Vision Transformers

This paper first answers the question "why do the two most powerful
techniques Dropout and Batch Normalization (BN) often lead to a worse
performance when they are combined together?" in both theoretical and
statistical aspects. Theoretically, we find that Dropout would shift the
variance of a specific neural unit when we transfer the state of that network
from train to test. However, BN would maintain its statistical variance, which
is accumulated from the entire learning procedure, in the test phase. The
inconsistency of that variance (we name this scheme as "variance shift") causes
the unstable numerical behavior in inference that leads to more erroneous
predictions finally, when applying Dropout before BN. Thorough experiments on
DenseNet, ResNet, ResNeXt and Wide ResNet confirm our findings. According to
the uncovered mechanism, we next explore several strategies that modifies
Dropout and try to overcome the limitations of their combination by avoiding
the variance shift risks.

本文通过理论和统计分析探讨了为何结合 Dropout 和 Batch Normalization 会导致神经网络预测准确率下降的问题，发现该组合会引起特定神经元的方差转移并导致预测不稳定，提出了一些修改 Dropout 的策略来避免方差转移的风险。