Vision Transformer (ViT) is emerging as the state-of-the-art architecture for
image recognition. While recent studies suggest that ViTs are more robust than
their convolutional counterparts, our experiments find that ViTs trained on
ImageNet are overly reliant on local textures and fail to make adequate use of
shape information. ViTs thus have difficulties generalizing to
out-of-distribution, real-world data. To address this deficiency, we present a
simple and effective architecture modification to ViT's input layer by adding
discrete tokens produced by a vector-quantized encoder. Different from the
standard continuous pixel tokens, discrete tokens are invariant under small
perturbations and contain less information individually, which promote ViTs to
learn global information that is invariant. Experimental results demonstrate
that adding discrete representation on four architecture variants strengthens
ViT robustness by up to 12% across seven ImageNet robustness benchmarks while
maintaining the performance on ImageNet.

本文介绍了如何通过在 ViT 的输入层添加离散 token 的方法来提高其对全局信息的学习和对真实世界数据的泛化能力，并通过实验证明对于七个 ImageNet 鲁棒性基准测试，该改进方法能够让 ViT 的鲁棒性提高最多 12%，同时不影响在 ImageNet 上的性能表现。