Ensembling has proven to be a powerful technique for boosting model
performance, uncertainty estimation, and robustness in supervised learning.
Advances in self-supervised learning (SSL) enable leveraging large unlabeled
corpora for state-of-the-art few-shot and supervised learning performance. In
this paper, we explore how ensemble methods can improve recent SSL techniques
by developing a framework that permits data-dependent weighted cross-entropy
losses. We refrain from ensembling the representation backbone; this choice
yields an efficient ensemble method that incurs a small training cost and
requires no architectural changes or computational overhead to downstream
evaluation. The effectiveness of our method is demonstrated with two
state-of-the-art SSL methods, DINO (Caron et al., 2021) and MSN (Assran et al.,
2022). Our method outperforms both in multiple evaluation metrics on
ImageNet-1K, particularly in the few-shot setting. We explore several weighting
schemes and find that those which increase the diversity of ensemble heads lead
to better downstream evaluation results. Thorough experiments yield improved
prior art baselines which our method still surpasses; e.g., our overall
improvement with MSN ViT-B/16 is 3.9 p.p. for 1-shot learning.

本文使用数据相关加权交叉熵损失函数的框架，探索如何通过集成方法提高最近的自监督学习技术，实验证明我们的方法在 ImageNet-1K 的少样本学习中优于现有的 SSL 方法