Contrastive methods have led a recent surge in the performance of
self-supervised representation learning (SSL). Recent methods like BYOL or
SimSiam purportedly distill these contrastive methods down to their essence,
removing bells and whistles, including the negative examples, that do not
contribute to downstream performance. These "non-contrastive" methods work
surprisingly well without using negatives even though the global minimum lies
at trivial collapse. We empirically analyze these non-contrastive methods and
find that SimSiam is extraordinarily sensitive to dataset and model size. In
particular, SimSiam representations undergo partial dimensional collapse if the
model is too small relative to the dataset size. We propose a metric to measure
the degree of this collapse and show that it can be used to forecast the
downstream task performance without any fine-tuning or labels. We further
analyze architectural design choices and their effect on the downstream
performance. Finally, we demonstrate that shifting to a continual learning
setting acts as a regularizer and prevents collapse, and a hybrid between
continual and multi-epoch training can improve linear probe accuracy by as many
as 18 percentage points using ResNet-18 on ImageNet. Our project page is at
this https URL.

通过对 SimSiam 两个非对比损失方法进行实证分析，研究发现 SimSiam 对数据集规模和模型大小非常敏感，并提出用崩溃度量来衡量降维崩溃的程度，进而预测下游任务性能。同时，采用持续学习模式作为规范器可以预防模型的崩溃，并提出一种持续和多轮训练的混合方式，该方式能够在 ImageNet 上使用 ResNet-18 显著提高线性探针精度。