We study semi-supervised learning (SSL) for vision transformers (ViT), an
under-explored topic despite the wide adoption of the ViT architectures to
different tasks. To tackle this problem, we propose a new SSL pipeline,
consisting of first un/self-supervised pre-training, followed by supervised
fine-tuning, and finally semi-supervised fine-tuning. At the semi-supervised
fine-tuning stage, we adopt an exponential moving average (EMA)-Teacher
framework instead of the popular FixMatch, since the former is more stable and
delivers higher accuracy for semi-supervised vision transformers. In addition,
we propose a probabilistic pseudo mixup mechanism to interpolate unlabeled
samples and their pseudo labels for improved regularization, which is important
for training ViTs with weak inductive bias. Our proposed method, dubbed
Semi-ViT, achieves comparable or better performance than the CNN counterparts
in the semi-supervised classification setting. Semi-ViT also enjoys the
scalability benefits of ViTs that can be readily scaled up to large-size models
with increasing accuracies. For example, Semi-ViT-Huge achieves an impressive
80% top-1 accuracy on ImageNet using only 1% labels, which is comparable with
Inception-v4 using 100% ImageNet labels.

该研究探讨了图像 Transformer（ViT）的半监督学习（SSL），提出了一种新的 SSL 流程和一种概率伪混合机制，Semi-ViT 可以实现与卷积神经网络相媲美或更好的性能，并且具有可扩展性。