Vision Transformer (ViT) has emerged as a prominent architecture for various
computer vision tasks. In ViT, we divide the input image into patch tokens and
process them through a stack of self attention blocks. However, unlike
Convolutional Neural Networks (CNN), ViTs simple architecture has no
informative inductive bias (e.g., locality,etc. ). Due to this, ViT requires a
large amount of data for pre-training. Various data efficient approaches (DeiT)
have been proposed to train ViT on balanced datasets effectively. However,
limited literature discusses the use of ViT for datasets with long-tailed
imbalances. In this work, we introduce DeiT-LT to tackle the problem of
training ViTs from scratch on long-tailed datasets. In DeiT-LT, we introduce an
efficient and effective way of distillation from CNN via distillation DIST
token by using out-of-distribution images and re-weighting the distillation
loss to enhance focus on tail classes. This leads to the learning of local
CNN-like features in early ViT blocks, improving generalization for tail
classes. Further, to mitigate overfitting, we propose distilling from a flat
CNN teacher, which leads to learning low-rank generalizable features for DIST
tokens across all ViT blocks. With the proposed DeiT-LT scheme, the
distillation DIST token becomes an expert on the tail classes, and the
classifier CLS token becomes an expert on the head classes. The experts help to
effectively learn features corresponding to both the majority and minority
classes using a distinct set of tokens within the same ViT architecture. We
show the effectiveness of DeiT-LT for training ViT from scratch on datasets
ranging from small-scale CIFAR-10 LT to large-scale iNaturalist-2018.

利用 DeiT-LT 方案能够在长尾数据集上从头开始训练 ViT，并使用蒸馏 DIST 令牌通过在 ViT 架构中使用不同的令牌来学习对应于多数和少数类的特征。